Aptana will build off of its IDE and AJAX success with its new technology. Read the full story.
Learn more about Aptana Cloud, an "Elastic Application Cloud" that's ideal for Web developers who use scripting languages.
Join the early access program.

One of the things I was working on recently is 'screen scraping' some data from several sites and presenting them in a single page. Of course Jaxer seems like the perfect tool for this as I can go fetch the data server-side, pull it all together, then present the final page to the browser. It was so exciting to see this work that I thought I'd make this post to share my experience with you!

We’re working now on adding support in Jaxer for being able to programmatically create 'window' objects, filling them with content from a remote URL, having that content actually execute, then being able to go into that window object and pull DOM elements out.

Unfortunately for me, that work isn't ready yet, so I wanted to see if there was some workaround I could use until it was ready. Well, after some research and trial and error, a workaround was found. The only catch with this code is that if there is any JavaScript on the page(s) you are fetching, that code will not execute, that’s coming soon.

Note that all of the code below can be contained within a single script tag:

<script runat='server'>...all code here...</script>

So, the first thing I wanted to do is make it really easy to fetch a remote document and grab out some content. I didn't want to use string matching or regular expressions of any kind, I wanted to use my trusted 'getElementById' to fetch named elements directly. (Of course I ran into sites that don't name their elements, but fortunately many use named classes, so I had to fetch them that way, but I get ahead of myself.)

Let's get started. The following two lines is all that is required to fetch a remote URL synchronously on the server and then pull out a named element:

var doc = getDocumentFromURL("http://www.news.com/"); var item = doc.getElementById("mostPopStories");

Now that we have 'item' as the element we were trying to fetch, we just have to add that element to our current blank page. I created a simple function called 'addElementToPage(title, element)' that takes care of this for me. Here's the next line:

addElementToPage("News.com Popular Stories", item);

That's it. Now I do that 2 more times to fetch content from 2 other sites.

doc = getDocumentFromURL("http://ajaxian.com"); item = getElementsByClass('activitystream', doc)[0]; addElementToPage("Ajaxian Activity Stream", item); doc = getDocumentFromURL("http://www.washingtonpost.com/"); item = getElementsByClass("top-box-in", doc)[0]; addElementToPage("Washington Post", item);

You'll notice that I didn't use getElementById() as those elements were not named, but fortunately they had class names associated with them. I found a convenient function called getElementsByClass() that I used in this case. Since it returns an array of elements, I use the [0] index to retrieve the first item in the list, which in my case, is probably the only item.

Believe it or not, that's basically it. At this point, your three 'DOM fragments' have been fetched and inserted into your new document. Here's the result:

Following are the convenience functions that help make the magic happen. The first function is addElementToPage(). It very simply just creates an H2 tag and sets the title and a DIV tag which contains the contents of the fetched element:

function addElementToPage(title, element) { var el = document.importNode(element, true); var h2 = document.createElement('h2'); h2.innerHTML = title; document.body.appendChild(h2); var div = document.createElement('div'); div.appendChild(el); document.body.appendChild(div); }

This next function, getDocumentFromURL(), is the one that does most of the work. (I found some good info on the subject of HTML to DOM here: http://jszen.blogspot.com/2007/02/how-to-parse-html-strings-into-dom.html) It first goes and retrieves the remote page. Then it creates a 'document fragment' from the contents of the fetched site. That fragment is then added to a dynamically created IFRAME. Finally, the Document object from the IFRAME is fetched and returned. In short, we can pass in a URL, get the string value, place it into an IFRAME, then pull out the resulting Document object so that we can work on it.

function getDocumentFromURL(url) { // Get remote site var string = Jaxer.Web.get(url); // Parse the string into a doc fragment var range = document.createRange(); range.selectNode(document.body); var parsedHTML = range.createContextualFragment(string); // Create the iframe that will contain the site we loaded var iframe = document.createElement("iframe"); document.body.appendChild(iframe); // Import from doc fragment into the iframe var el = iframe.contentDocument.importNode(parsedHTML, true); iframe.contentDocument.body.appendChild(el); var newDoc = iframe.contentDocument; // delete the iframe as we don't need it any more document.body.removeChild(iframe); return newDoc; }

This final function was found at http://www.dustindiaz.com/getelementsbyclass/ and walks a node looking for an element with a specific class name. It was used in the case where elements don't have ID's, so a class name is used instead.

function getElementsByClass(searchClass, node, tag) { var classElements = new Array(); if (node == null) node = document; if (tag == null) tag = '*'; var els = node.getElementsByTagName(tag); var elsLen = els.length; var pattern = new RegExp("(^|\\\\s)" + searchClass + "(\\\\s|$)"); for (i = 0, j = 0; i < elsLen; i++) { if (pattern.test(els[i].className)) { classElements[j] = els[i]; j++; } } return classElements; }

What's exciting about this sample is that it is relatively simple, uses the full power of server-side JavaScript and more importantly, Jaxer's cool server-side DOM capability to enable real 'DOM scraping'. Once window object creation is finished in Jaxer (real soon now), then you'll be able to fetch remote pages, execute their integrated code, then proceed to fetch out items from the resulting DOM.

Tony Issakov has written up a nice conceptual article on how Aptana Jaxer can simplify certain tasks of dojo and dijit development. He starts with a simple case of how Jaxer can enable simplified server-side dijit markup, then expands the ideas to JSF, JSTL, and templates. Tony explains: "Once upon a time dojo had markup ability using custom tags. This was pulled because as it turns out, some browsers just scrap tags they don't understand. One browser that did support tags of a non HTML nature though was Mozilla and in turn we get this freedom back via Jaxer." For example: <div dojoType="dijit.form.button" label="test"></div> can become... <dijit type="button" label="test2"></dijit>

Aptana Jaxer has been updated to 0.9.5 providing increased performance for server-side JavaScript and a host of other enhancements to make End-to-End Ajax development easier.

Aptana Studio users can get the latest via Help > Software Updates. Otherwise, download the latest now.

• Increased Performance
  • Aptana Jaxer's core is now based on Mozilla FireFox Beta3 and thus delivers improved JS performance and memory management.
  • Jaxer now pre-parses scripts to bytecode for faster execution during callbacks.
• More Natural JS Environment
  • Window and document objects on the server-side now parallel more closely how you work with them on the client-side. This means a more natural coding style with server-side JavaScript in Jaxer and a more browser compatible experience.
  • Jaxer no longer inlines external scripts to the page, this allows for correct use of browser cache on client side content and makes 'view source' a much more pleasant experience. This also applies to Jaxer's ClientFramework (the client-side JavaScript Jaxer puts in pages to facilitate callbacks and remote method invocation). Instead such external scripts are cachebusted with the buildnumber so that upgrading the server will automatically expire any old client side cache copies.
  • A new Autoload feature has been added to allow for a script to be cached so that during callbacks the script does not have to be loaded on demand (thus faster) and the previous oncallback function used to setup the callback environment is no longer needed for library loading. A configuration option is provided to expire the script every time the containing page is accessed. This is allows for simpler development and is the configuration with which Jaxer show ships.
• Improved Platform Support
  • Significantly improved Unix support. Jaxer now builds and runs on Solaris too!
  • mySQL support is now also provided for Mac OSX
  • A number of Windows Vista security/UAC related problems have been addressed
• API Enhancements
  • Jaxer's file system API has been updated -- mostly around the 'quick' static method. For example: Jaxer.File.chmod('myfile.tx',0400)
  • A simple Stopwatch timer API now allows for timing measurements in code.

One of the common web tasks that always seems to involve a lot of arcane knowledge of backend systems, is how to upload files. Like most systems, Aptana Jaxer uses the post/receive model, where a web form is posted to the server, and the target of that form will receive and process the content. Where Jaxer makes this really easy is that it is all done with regular JavaScript and HTML. No special enviroment vars, excessive string processing, or finding handles to oddly named temp files is required.

All you need is a form to present to the user, which contains an input element of type upload, and an HTML page to receive the submitted form containing a "runat=server" script block that retrieves the data from the request. You could have the form and the recipient be the same page, however in this example I'll use two files for clarity. As an aside in Jaxer you can actually send the form directly to a JavaScript function living on the server but we'll look at that configuration in a future blog entry.

Example HTML Upload Form

This example upload form simply allows you to select up to 2 files for upload from the local filesystem. Pressing 'upload' posts the contents of that form to an HTML page containing the JavaScript shown later.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> <title>Upload </title> </head> <body> <form id='upload' action="submissions.html" method="post" enctype="multipart/form-data"> <input type="file" name="myFile1"><br> <input type="file" name="myFile2"><input type="submit" value='upload'> <p> 'upload' sends the selected file(s) to the server. </p> </form> </body> </html>

The form will look like the following when viewed in a browser. Pressing the browse button will present you with a file selection dialog, or you can type the path into the input box directly. I've used 2 file slots in this example to demostrate that the files passed in are available as part of an files array contained in the request object (a JavaScript representation of the HTTP request) and are accessed like any other JavaScript array.

Example Aptana Jaxer code to receive file uploads

To receive the data from the form when submitted we put some Jaxer code into the page the form will be submitted to. The code below should be in script block with a runat = 'server' attribute, which makes the code run serverside and doesn't present it to the client so you don't expose any serverside filenames or folder structures.

<script type='text/javascript' runat='server'> var message = ""; for (fileCount=0; fileCount<Jaxer.request.files.length; fileCount++) { var fileInfo = Jaxer.request.files[fileCount]; var destinationFilePath = Jaxer.Dir.resolvePath(fileInfo.originalFileName); fileInfo.save(destinationFilePath); message += "<br>" + [ "Saved to : " + destinationFilePath , "original filename : " + fileInfo.originalFileName , "temp filename : " + fileInfo.tempFileName , "contentType : " + fileInfo.contentType , "size : " + fileInfo.fileSize ].join("<br>"); } document.write(message); </script>

The data posted by the form is available within the Jaxer.request object.

For the purposes of this example we are specifically interested in the Jaxer.request.files array which contains an array of Jaxer.Request.FileInfo objects, one for each file posted with the form.

The Jaxer.Request.FileInfo object contains some useful properties describing the file, such as fileName, contentType, etc.

Uploading two files from my filesystem from the form in this example would return a page of output like below. We have uploaded the file to same location as the submission.html file used as the form action in the originating HTML.

Saved to : C:\aptana\JaxerDev\public\work\bar.txt
original filename : bar.txt
temp filename : C:\aptana\JaxerDev\tmp\tmp
contentType : text/plain
size : 15754
Saved to : C:\aptana\JaxerDev\public\work\foo.txt
original filename : foo.txt
temp filename : C:\aptana\JaxerDev\tmp\tmp-1
contentType : text/plain
size : 816

Well that's really all there is to it. It's simple and easy and you never have to think about anything other than HTML and Javascript.

API docs for the FileInfo object and the Jaxer.Request object are available online.

Next week Aptana will be at AjaxWorld (NYC), iPhone Developer Summit (NYC), and EclipseCon (Silicon Valley). Join us. We want to meet you, answer your questions, and get your feedback and ideas to make Aptana Studio and Aptana Jaxer even better for you. (Of course you can do that anytime via our ASAP enhancement request system too).

AjaxWorld, March 18-20, New York, NY
Kevin Hakman presents Aptana Studio: Your Unfair Advantage for AJAX, iPhone, Adobe AIR, PHP and Rails Development and Rapid Development of Enterprise Ajax Apps both on Wednesday, March 19.

iPhone Developer Summit, March 18-20, New York, NY
Also on Wednesday March 19, Kevin Hakman presents Developing AJAX Applications for iPhone and iPod Touch.

EclipseCon, March 17-19, Santa Clara, CA
Eclipse Monkey committer and Aptana engineer Ingo Muschenetz shares Building Eclipse-based Products: A View From the Trenches on March 18, and Aptana engineer Kevin Sawicki delivers Pimp My Editor with insights for extending and customizing features in Eclipse.

We'll also be at the EclipseCon Ajax focused Birds of a Feather meeting -- check the schedule for details.

Yesterday Adobe announced the availability of Adobe AIR 1.0. At the same time, Aptana released the Adobe AIR plugin for Aptana Studio which simplifies development of Ajax applications that run "on AIR". (Adobe AIR Plugin for Aptana Studio)

If you are not familiar with Adobe AIR, it enables you to use the skills and Web technologies you already know, HTML, CSS, and JavaScript, to write applications that can be deployed to the Windows and Mac desktop, and shortly on Linux desktops too.

And what is Aptana Jaxer, you ask? Jaxer is an Ajax server that enables you to leverage those same Web skills and technologies, HTML, CSS, and JavaScript, to write server-side code and do a whole new range of things with JavaScript like interact with databases, file systems, and remote data sources, do server-side DOM manipulations, cross-domain data calls, make socket connections, and call server-side JavaScript functions from the client-side, plus lots more. Yes, you write both client and server code using HTML and Ajax. You can even write entire web applications in a single HTML file if you wish.

With that said, it becomes much clearer as to what ‘Jaxer on AIR’ is all about -- two systems that enable you to leverage your Ajax skills for building desktop and server apps, unified into a single, very interesting model. With Jaxer you can implement your server-side to know if an app is running in AIR and thus take advantage of offline synchronization and local access to the client system as permitted, or if you’re running on the Web with a subset of the total potential features of your Ajax apps and Web pages.

I did a 10-minute screencast showing a simple example of this. I take one of Jack Slocum’s great Ext / Adobe AIR demos (EXT Tasks Demo) and Jaxer-enabled it. I added a single JavaScript function that can query my backend database, and I use the results to fill the user interface.

This demo just scratches the surface of what is possible, but I hope it opens the doors to your thinking about how to utilize these great technologies – it’s my version of their chocolate and our peanut-butter happily together, enjoy!

View 'Jaxer on AIR' Screencast

This past week we released another point release of Jaxer, version 0.9.3, updating the standalone server as well as the one packaged within Aptana Studio. Of course there were bug fixes, performance improvements, and API enhancements. Perhaps most important, this was our first release to officially support Linux. Supporting Linux isn't trivial because of the multitude of variants, but it is very strategic as the deployment platform of choice for many people. Look for more distros being supported and more forms of distribution, from buildable source drops to tarballs and installable packages and even to complete EC2 and VMWare images. And thanks to the community for already helping us with supporting Linux distros.

So where are these releases leading — what are some things on the immediate roadmap for Jaxer?

• An even more Ajax-y experience: the callback environment will be made more natural, server-side support for Ajax libraries will expand, mashups will be easier to implement, etc.
• Scalability and performance: we'll release some interesting benchmarks on current Jaxer performance and stability, illustrate how it scales, and make some significant improvements.
• More APIs: we'll make database, file, network, and native interfaces more powerful while maintaining clean APIs that feel at home in JavaScript and HTML.
• More integration: we're creating a true bridge to Java by leveraging DWR (Direct Web Remoting) server-side, in addition to our current support for Tomcat (and other servlet containers); we'll expand the list of supported web servers from the current Apache 2.x and Jetty to IIS, Apache 1.3, and perhaps others; and we'll work on other database drivers.
• Deployment: we're working on making one-click deployment a reality, so you can develop your Ajax apps in Studio, leveraging Jaxer for the server side, and click to deploy your app to a hosted, managed, monitored, and scalable environment.
• Tooling: How about seamless end-to-end debugging of your Ajax app — step through your program starting on the server, move to the browser, go back to the server for a callback, then back to the browser for processing the result...
• Real-world apps: in the spirit of eating our own dog food, and supporting dog food consumption in the community, look for complete applications written on Jaxer and en expansion of our web site and of Studio so you can share your snippets, samples, projects and applications.

Oh, and the time frame? We're not exactly the patient type, so for many of these think weeks rather than months. Want something specific to come out even faster? Join the discussions on the forums, join the effort (some folks are supporting Linux distros, others are building persistence frameworks, some are putting together samples), or join our team.

Are you looking for a quick and easy way to deploy your Jaxer applications? You can now take advantage of the cloud technology provided by Amazon's Elastic Compute Cloud (also known as EC2). EC2 provides you with a fast way to get going with Jaxer and to deploy it on a hosted infrastructure. With your own EC2 instance, you can make your Jaxer applications available for both private and public consumption.

The Jaxer Team has set up a CentOS image with Jaxer pre-configured and ready to go. Find out how to get your own instance running (and more) on the Running an EC2 Instance of the Jaxer Server page.

We'd love to hear about what you're building with Jaxer, so if you're working on a Jaxer app or just have a few questions about Jaxer, we'd like to invite you to participate in our Jaxer community forums.

The Jaxer Team is proud to announce that we now support Jaxer for several Linux platforms:

• Ubuntu
• CentOS
• Fedora

To learn how to get up and running on Linux, see the Jaxer Standalone Installation guide for Linux.

We've been working with volunteer beta testers within the Jaxer community to bring you these distros, and we plan to support additional distros in the future. However, this is our initial release for Linux support and because the various Linux platforms have so many nuances, some Linux platforms could likely still have some issues. Our Linux support is very community driven, so we invite you to participate in the platform vote on the forums to voice your opinion about your preferred platform:

http://forums.aptana.com/viewtopic.php?t=4844

If you have Jaxer up and running on a platform that we have not listed, let us know so we can add it to the list of supported platforms. Additionally, if you need help troubleshooting issues with Jaxer on your Linux platform, post your questions on the Jaxer forums.