JRuby 1.0 Released!

We have finally released JRuby 1.0, based on the last release candidate, RC3. And what more is there to say? Not really a whole lot...It's almost entirely RC3, with one or two minor fixes added in. But it's really turned out to be an outstanding release, and already reports are coming in of folks trying it out en masse. We're very happy.

So I'll do a little recap here. JRuby 1.0 was focused almost entirely on one goal: Ruby 1.8.x compatibility. To that end, we are now the only alternative Ruby implementation that can reasonably claim we're "compatible". It's no longer a question of whether we can run Ruby applications or not...we've proven that again and again. The issues people run into now are those requiring minor behavioral tweaks, minor parser tweaks, and occasionally exploration of some peculiar threading or memory concerns. It's been a long time coming, but the compatibility issue is largely answered.

Now we start looking toward the future. Once you have a working Ruby implementation, what next? I believe we've shown that the correct way to approach Ruby is to get it right first. The next step is making it run as well as possible. Ok, so we've cheated a bit along the way, introducing interpreter optimizations and a JIT compiler, but that's all fun stuff. The heavy lifting for performance and scalability are coming up, and there's a lot of low-hanging fruit we can start to tackle with 1.0 safely behind us. So action item #1 for the future of JRuby is plucking that low-hanging performance fruit.

The second item for any working implementation would have to be platform integration. Our platform is Java (and of course we've cheated a bit here by doing additional work to make Java integration really useful and usable), so integration involves making Ruby a better citizen of the Java platform. In this area, expect to see us further reduce the disconnect between Ruby and Java, allowing you to construct Ruby objects directly from Java code, define real Java classes using Ruby (classes you can then compile Java code to call), and minimizing to as large an extent possible the performance impact of calling from dynamically typed Ruby code into statically typed Java code. Java integration is action item #2.

And then what? Well, there are many options, all very attractive. I personally would like to see time spent implementing potential Ruby 1.9/2.0 features, to provide a second testbed where people can try those features out. I would like to find ways to share code with Rubinius (beyond tests), either by implementing just enough Rubinius logic to run its pure-Ruby core-class implementations or by implementing a full Rubinius kernel on the JVM. I would like to see JRuby expand to the rest of the Java world, bringing the Ruby Way to Java EE. But most importantly, I want to reach out to the Ruby community and find ways for them to be a part of the JRuby process.

Up to now, JRuby has really existed in its own, separate world. There was the Ruby community and also the JRuby community, and although members might identify themselves with both, there was always this perceived boundary.

We need to break that boundary down...not only for JRuby but for the other Ruby implementations as well.

Ruby is coming of age. Multiple implementations shows that Ruby has really matured as a language...and also shows it has a lot of maturing left to do. Ask Evan (of Rubinius) or me how we feel about retry behavior or block argument processing or thread event processing or SAFE levels and tainting and you'll start to understand some of the ugly, hidden bits of Ruby. We need to make sure that Ruby development proceeds as a whole...not necessarily as a single project or a single codebase, but as an open, direct exchange of concerns, complaints, solutions and ideas. We need to start treating the Rubinius community and the JRuby community and any other implementations' communities as part of the whole...different facets of the same gem.

If you have never tried an alternative implementation of Ruby: do it today. Pick out your favorite app, library, or framework and build your next app using something other than Matz's Ruby. Start running your continuous integration tests against JRuby trunk (or Rubinius trunk, if it runs your code ok; those guys sorely need more CI hits). Make sure your gem releases work on JRuby too, and if you have native (i.e. C) code, explore what it would take to do a JRuby port. Show that you welcome diversity into the Ruby world...that you recognize that diversity is an essential part of language evolution.

JRuby has always been a community project, and only by absorbing the JRuby community into the Ruby community will JRuby continue to be successful. If we can make that happen, I see wonderful things in Ruby's future...regardless of which implementation you use.

A Response to Ola's IronRuby Post

I'm not up for creative titles tonight. Hopefully you'll see this in your feed reader and click for a second opinion. Granted, I agree with Ola's IronRuby post on most points, but I disagree on a few key items. So let's dive in, shall we?

You managed to blog this viewpoint before me, Ola, but you know I agree with almost everything. However, to pour a bit more oil on the fire, I'll take it a step further.

Having run the Ruby gauntlet and brought JRuby from not running anything to running Rails almost 100% perfectly (in just over a year, I might add), I will confidently say there's no way with current specs and tests that anyone could create an implementation of Ruby from scratch that will run Rails unless they can look at the existing implementations. I simply do not believe it's possible.

And I'll throw another couple curve balls too:

• I don't believe tests and specs will be where they need to be within the next 6-12 months unless there's a major effort put behind them. Even if that effort happens, I still don't think we'll ever have specs enough for someone to implement Ruby "in the dark" for at least a year, if it ever happens.
  
• IronPython did a great job getting pretty close to 100% compatible. But Jim Hugunin had implemented an almost 100% Python-compatible implementation in Jython before going to Microsoft; he didn't need to look at the Python source. I don't believe John Lam has the same level of experience with Ruby, so he's at a severe disadvantage (and to John: I really feel for you, man...this has got to be difficult).
• This is a good friend's belief, but he's won me over: we don't believe Microsoft would ever willingly allow IronRuby to get to the point of running Rails, since that would directly compete with their ASP.NET server, software, and tool offerings. What would be the benefit to them of a free runtime running a free language implementation that runs a free web framework? Probably zero. And as Martin Fowler and others have blogged about NUnit, Microsoft hasn't exactly been lovey-dovey with OSS projects that impact (or are perceived to impact) their bottom line.
• Given these facts and the current situation, I'd say it's a better bet for us as a community to get behind the Gardens Point Ruby.NET Compiler project, which is already much farther along than IronRuby...plus it's real open source (you can contribute) and they can look at Ruby's source (and have admitted to doing so for at least the parser). I was wary/skeptical of Ruby.NET last year, but they now seem like the current best hope for Ruby on the CLR. That link again is Gardens Point Ruby.NET Compiler. And to the QUT guys working on Ruby.NET, you need to do three things right now to save your project from historical obscurity: set up a public source repository, accept a few external contributors, and start blogging and emailing up a freaking storm.

That about sums up what I have to say about IronRuby, Microsoft, and the future of Ruby on the CLR. The rest of Ola's points I agree with...diversity is important, a spec and test kit are needed immediately, and Microsoft needs to change their OSS attitude pretty quick or risk becoming irrelevant.

JRuby 1.0.0RC3 Released - And This Is It!

Tom posted the announcements already, but JRuby 1.0.0RC3 is out in the wild! This release is our most important yet, because we intend for this release to become JRuby 1.0. The only things that will change from now until a 1.0 final release later this week would be any showstopping bugs that are extremely low-impact to fix. In general, RC3 should be nearly identical to 1.0 final.

So what does this mean for JRuby? We've taken the approach of calling the JRuby 1.0 release the "Ruby compatible" release. Basically, all known application bugs caused by JRuby incompatibilities with Matz's Ruby (MRI) have been resolved. This doesn't mean there aren't more compatibility bugs; there probably always will be, and we have probably 50-80 in the bug repository right now (and another 80 bugs or enhancements that are JRuby-specific). But it does mean that most applications should "just work" out of the box, and any application failures we knew of have been resolved. In general, the remaining post 1.0 bugs are edge cases and minor correctness issues, like obscure forms of core methods or missing error conditions.

We've tried very hard to make this both a solid release and a stepping stone to JRuby's future. With this release, we can confidently say that we're Ruby 1.8.5 compatible. Any additional fixes we need to make (like the known issues) are in the last 1% of Ruby features we can potentially support. We also recognize that this is a big moment for JRuby. People can start counting on JRuby to run their Ruby applications correctly. Of course many in the JRuby community have already been doing this for many months, but the 1.0 moniker says to the world we feel like we're ready for prime time.

So what comes after 1.0? That breaks down into a few areas:

1. Performance. JRuby's performance has come a long way in the past year. We've increased speed by at least an order of magnitude, and have enabled the JIT compiler for the 1.0 release. This means that for many cases where JRuby can compile Ruby code, it will perform faster than Ruby 1.8.5. The general case is still a little muddy, though. We've had anecdotal reports that JRuby on Rails in a Java application server performance extremely well, and other reports that general Ruby applications perform somewhat poorly. The general performance situation is not well-understood, but we all agree there's a lot more work to be done. And the good news is that we have a big list of optimizations remaining to continue improving JRuby's specific and general performance.
2. Java Integration. JRuby does an excellent job of fitting into the Java platform. In almost all cases, you can call libraries, implement interfaces, and extend classes with no difficulty. But there are edge cases--usually nonstandard or antipattern Java--that JRuby doesn't behave as nicely. The code to enable calling Java is also far more complicated than we'd like. To solve both issues, we'll be taking the existing Java integration syntax and API and backing it with a redesigned library. Expect to see something of this work in a 1.1 release later this year. Our goal is to achieve excellent integration with Java, on par with the most tightly-integrated JVM languages available today. And we're not far off.
3. Ruby 2.0 and Rubinius. We intend to start supporting Ruby 2.0 and Rubinius bytecode execution soon, though the Ruby 2.0 work is much farther along. We intend to catch up the Rubinius work as well as to try adopting some of Rubinius's pure Ruby implementations of core class functionality. We also plan to enable the use of Ruby 2.0 features through configuration and command-line switches to JRuby. Specify something like -J-Djruby.string.version=2 and all strings in the system will be treated as Ruby 2.0 strings, with full character semantics. Or optionally turn on experimental Ruby 2.0 syntax for lambdas and named parameters. Basically we want to bring JRuby up to the bleeding edge of Ruby features, to provide another platform where people can get a feel for what Ruby 2.0 might look like. More exposure for these features will help Matz and Co. decide the best way to move forward. And that's good for everyone.

And of course we'll continue fixing compatibility bugs, Java integration bugs, and expanding JRuby's reach on the Java platforms with more DSLs and API wrappers around all your favorite Java APIs.

But there's one more area I should talk about: how you can get involved.

JRuby is a community project. The core committers play traffic cops as much as developers, routing patches, examining bugs, documenting features. The only way a project like JRuby succeeds is through mass community involvement. We've been extremely lucky to have a constant flow of interested developers into our community, even when OSS community attrition has been very high. The community may look completely different from month to month, but the flow of patches and bugs has steadily increased.

For folks just getting started, I've written a few articles on the JRuby wiki that should help you understand the development process. They're linked from the front page under "Getting Involved".

For easy bugs, I'd recommend looking in the JRuby JIRA for bugs with "rubinius" or bugs reported by Daniel Berger. The Rubinius bugs are almost all problems we've had running Rubinius's excellent specs, and usually represent minor incompatibilities with C Ruby. Daniel's tests are a lot of those edge cases I mentioned...he's running through his own test suite and reporting any incorrect behavior that's come up. Another option would be to just pick your favorite Ruby app and start running its test cases with JRuby. You're sure to find something interesting to report, and it may be an easy fix.

For non-Java-coders (or folks afraid of hacking on JRuby), stop by the RubySpec Wiki and update or author an article. RubySpec is an effort to build a community-driven specification of Ruby that all users and implementers can freely reference. It is linked from the RubyDoc site and is fast becoming a standard way for the community to record language and library behaviors. I believe this is the best and fastest way for us to form a complete specification of Ruby's behavior...and I believe such a specification is becoming extremely important, what with there now being 5-10 different implementations of Ruby, all guessing at what "correct" is. Have you written RubySpec today?

Well that's about it for this release report. If all goes as well as I think it will, JRuby 1.0 final should be released some time this week. Give it a try, I think you'll be pleasantly surprised!

Creating a Field-Initializing 'new' Method

One thing often touted as a missing feature in Ruby is the lack of a constructor form that initializes fields. A few other languages have this feature, including for example Groovy, another JVM dynamic language. The general idea is that if you want to construct an object and initialize a number of fields, you often want to do it in one shot. Rather than modify the class to have additional initializers for all the fields you want to set, there's another option.

Because Ruby is so cool, you can add this feature yourself to all classes at the same time.

class Class
  def new!(*args, &block)
    # make sure we have arguments
    if args && args.size > 0
      # if it's not a Hash, perform a normal "new"
      return new(*args, &block) unless Hash === args[-1]

      # grab the last arg in the list
      last_arg = args.pop

      # make sure all fields actually exist
      last_arg.each_key {|key|
        unless public_instance_methods.include?("#{key}=") do
          raise ArgumentError.new(
                           "No attr setter for name: #{key}") 
        end
      }

      # create the object and set its fields
      new_obj = new(*args, &block)
      last_arg.each {|key, value|
        new_obj.send "#{key}=", value
      }
    else
      # no args, just do a normal "new" with any block passed
      new_obj = new(&block)
    end
    new_obj
  end
end

So with such a simple piece of code, we now have a new! method on all classes that accepts a final parameter--a hash of field names and values--that can be given using Ruby's named-parameter-like syntax. Given a simple class, like the following:

class MyObject
  attr_accessor :foo
  attr_accessor :bar
  
  def initialize(msg)
    puts msg
  end
end

No additional work is needed to use our new! method:

x = MyObject.new!("yippee",
                    :foo => "hello", :bar => "goodbye") 
 => "yippee"
p [x.foo, x.bar]
 => ["hello", "goodbye"]
y = MyObject.new!("blah", :yuck => "baz")
 => error: "No attr setter for name: yuck"

The reason this works is that all classes are instances of the Class class. So the MyObject class definition above is roughly equivalent to saying:

MyObject = Class.new {
  # class def logic here
}

This means that instances of Class, like MyObject, inherit methods defined on Class, like new!. Since all classes in the system are Class objects, all classes instantly gain a new! method.

This is a perfect example of why Ruby is such a powerful language, and why it's so easy in Ruby to use the coolest metaprogramming tricks. And it's a primary reason why frameworks like Rails have been able to do such amazing things. With a language that's this powerful and this easy, you can imagine what else is possible.

Are we having fun yet?

Adding Annotations to JRuby Using Ruby

I love how meta-programmable Ruby is.

JRuby doesn't support annotations because Ruby doesn't support annotations. So what! We can extend Ruby to add something like annotations:

class JPABean
def self.inherited(clazz)
  @@annotations = {}
end
def self.anno(annotation)
  @@last_annotation = annotation
end

def self.method_added(sym)
  @@annotations[sym] = @@last_annotation
end

def self.get_annotation(sym)
  return @@annotations[sym]
end
end

Given this, we can do things like the following:

class MyBean :sql => "SELECT * FROM stuff WHERE name = 'foo'"
def foo; end

anno :field => :hello_id
attr :hello

anno :field => :bar_in_table
attr_accessor :bar
end

And here's some output from the resulting class:

p MyBean.get_annotation(:foo)
p MyBean.get_annotation(:hello)
p MyBean.get_annotation(:bar=)

=>

{:sql=>"SELECT * FROM stuff WHERE name = 'foo'"}
{:field=>:hello_id}
{:field=>:bar_in_table}

So as you can see we really do have all the necessary requirements to annotate classes. Now what if we just had MyBean be a java.lang.Object extension and stuffed the annotations into the resulting generated class? We can already create real Java classes in this way, but with the above syntax and a little magic in the JRuby Java integration later, we've got annotation support in on Java classes too. This should enable things like Hibernate, JPA, and JUnit 4 to work with JRuby's Ruby-based classes. Or at least, I believe it to be possible. It just requires a little work to add annotation information to the resulting generated classes.

I've planted the seed here and on the JRuby dev list...let's see if it germinates.

JVM Languages: The Future

I've started a Google Group for all those interested in the future of alternative languages (i.e. not Java) on the JVM. A number of you knew this was coming, and I've already invited folks that expressed an interest in my RedMonk Unconference session at JavaOne. The rest of you are also invited.

Please plan to talk shop about language implementation strategies, pain points on the JVM, and what we can do to build a common set of libraries, frameworks, and patterns to ease and improve the Java platform's support for many languages. The time for action is now, my friends, and we have a wealth of information across all our language projects to draw from. We must come together to build a stronger base for everyone to stand upon.

I welcome you to the future.

Foo Camp 2007

I've been invited to this year's Foo Camp. Since it sounds like a great time and there's a bunch of other folks going I'd like to talk to, I'm accepting the invitation. My purpose posting this entry is to let other campers know I'll be there and try to hook up and chat a bit beforehand. So drop me a line...firstname.lastname@sun.com.

FYI, if any other campers are coming up from Oakland, I'm renting a car. My times are listed on the "Rides Offered" page.

I'm looking forward to a fun weekend!

The Final Bugs

We're within weeks of a final JRuby 1.0. We've pared down the bugs that we think we can or must fix for 1.0 and this email is a summary of the ones we need help with. So whatever time you can spare, please have a look and help resolve these.

In order of decreasing priority in JIRA:

JRUBY-820: Net::HTTP.get behaves differently form MRI, failing to get UTF8 properly
-and-
JRUBY-828: UTF-8 regular expressions aren't working

I believe these are both largely the same problem, and it's probably the most visible remaining bug we need to fix. Regular expressions in JRuby currently do not work well with unicode strings, both as incoming match strings and as the regular expression strings themselves. REXML uses /u regular expressions, which is why I believe these issues are closely related. Wes Nakamura has commented that he's looking into 828, but more eyes will help ensure this is fixed.
I believe these are true blocking bugs for 1.0, and I'm not comfortable doing a release if they are not resolved.

JRUBY-971: Make it clear which Java method maps to each equality method

I think we have subtle equality bugs remaining largely because we don't use a consistent naming convention for the Java implementations of Ruby methods like ==, ===, eql? and so on. We should make a quick sweep through the system and make sure all core JRuby code is using the same Java method names for all of these, and binding them in the same way. Note: This is actually the cause of some set_trace_func bugs, since the default === impl should actually invoke ==, causing trace events for both.

JRUBY-969: Add rake and rspec to standard JRuby distribution for 1.0

We have decided to ship both Rake and RSpec with JRuby 1.0, since they are both well-accepted and widely used (moreso for Rake, but RSpec has gained a lot of acceptance the past year). But the mechanism of including them is unclear. I do not want to commit installed gems to SVN; I would prefer to just install them as needed for testing and when building the JRuby distribution. Thoughts?

JRUBY-672: java.lang.Class representation of Ruby class not retrievable

This bug is basically just looking for a way to get at the actual java.lang.Class representing a Ruby-based extension of a Java class. There may have been an API added with Bill's work, or this could be simple to add.

JRUBY-914: JRuby's BigDecimal needs to round

Stu submitted a patch for this, but it unfortunately depends on behavior only present in Java 5 and higher. Since most of us are pretty unfamiliar with BigDecimal, we don't know of a good workaround to support it correctly in Java 1.4.

JRUBY-822: jruby fails to report process exit status correctly

I think Nick may have a grip on this one, but if anyone can offer suggestions as to why the exit codes are apparently shifted in this way, please let us know.

JRUBY-966: Various issues with LocalJumpError not being created early enough
-and-
JRUBY-767: next in an eval should produce a local jump error

These are likely going to be up to Tom and I since they involve pretty deep runtime/interpreter work. But we're also looking for any remaining known issues with LocalJumpError/JumpException that are still out there. I believe we've fixed all such externally-reported issues at the moment.

JRUBY-888: Make regular distro/build create the all-in-one jar by default

We waffle back and forth on this. What's the answer? An all-in-one jar by default, or on request?

JRUBY-873: ant test thread tests sometimes run forever

I managed to get a trace on this one. It appears that some of the tests cause a deadlock between stopping a thread and other operations. But I don't believe anyone's seen it in normal execution yet (or at least nobody's reported it). I'm probably the only one familiar enough with the threading subsystem to fix it, but I'd like to know if anyone's seen other issues.

JRUBY-98: allow "/" as absolute path in Windows
-and-
JRUBY-36: Dir['...'] incompatibilities between Ruby and accross platforms
-and-
JRUBY-61: IO CRLF compatibility with cruby on Windows
-and-
JRUBY-644: Windows line delimiter (\r\n) cause position errors

These blasted Windows pathing and newline bugs just seem to hang on forever. Nobody wants to fix them. There must be someone out there that can help resolve these, yes?

JRUBY-884: Create and consolidate extension and non-standard options

There are a number of configuration options for JRuby. Some have their own flags, like -O to disable ObjectSpace and -C to force compilation of a target script. But many others are only configurable via Java system properties. What we really could use from the community is some idea which settings are worth adding special flags for. We can handle the rest. My personal opinion is that the new -J flag that allows passing flags to the JVM is actually enough.

By my estimation, the rest of the bugs are either almost done or are trivial enough they could be punted to a post 1.0 release. But if we can get these issues knocked down soon (starting with those top couple), 1.0 is going to be an extremely solid release.