Jonathan Gnagy's Blog

Ever want to script the installation of an application that runs on IBM WebSphere Application Server? Well for all of you out there who do (and I know this is some pretty specific geek speak), there are a couple pieces that I just recently ironed out.

Increasing the Deployment Manager’s Max Heap

This is pretty easy to do in Jython in a scriptable manner:

1
2
3
4
5

# Increase the Max Heap of the DMGR
dmgrNode = "someserver-dm"
newHeap = "1024"
AdminTask.setJVMMaxHeapSize('[ -nodeName '+dmgrNode+' -serverName dmgr -maximumHeapSize '+newHeap+' ]')
AdminConfig.save()

Modifying a Port

Sometimes new app servers might have port conflicts with other app servers (or anything else on the machine):

1
2
3
4
5
6
7
8

# Fix a conflicting port
appServerName = "solrServer"
serverName = "someserver"
nodeName = "someserver-node1"
newPort = "9091"
AdminTask.modifyServerPort(appServerName, 
  '[-nodeName '+nodeName+' -endPointName ORB_LISTENER_ADDRESS -host '+serverName+' -port '+newPort+']')
AdminConfig.save()

Create a Cluster from an existing App Server

This is also pretty easy:

1
2
3
4
5
6
7

# Create the Solr cluster
clusterName = "SolrCluster"
appServerName = "solrServer"
nodeName = "someserver-node1"
AdminClusterManagement.createClusterWithFirstMember(clusterName, "APPLICATION_SERVER", 
  nodeName, appServerName)
AdminConfig.save()

Installing an Application into a Cluster

Seems almost too easy:

1
2
3
4
5
6

# Install the CacheMonitor application
clusterName = "SolrCluster"
AdminApplication.installAppWithClusterOption("CacheMonitor", 
  "/opt/IBM/WebSphere/AppServer/installableApps/CacheMonitor.ear", 
  clusterName)
AdminConfig.save()

I’ve got more stuff, but that’ll probably do for now.

I’ve heard people say great things Apache Ant and old faithful make, but I haven’t heard enough people talking about how wonderful rake is to work with. I’ve used rake for many simple and complex tasks, from importing YAML files into a Rails project, to managing LDAP users. Rake offers the full power of Ruby, but still offers a fairly simple DSL.

Here’s an excellent railscast about rake: http://railscasts.com/episodes/66-custom-rake-tasks.

Here’s some code that demonstrates creating a simple rake task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

task :mkdirs do
  # Let's create some directories
  ["lib", "conf", "src"].each do |dir|
    Dir.mkdirs(dir) unless File.directory?(dir)
  end
end

# This task requires the 'mkdirs' task
task :copy_source => :mkdirs do
  require 'ftools'
  # Copy the source into the src directory...
  ["main.rb", "classes.rb"].each do |file|
    File.copy("/path/to/sources/#{file}", "src/#{file}")
  end
end

These two simple tasks create some directories and copy some source files. It also demonstrates task dependencies and pure Ruby file operations.

I uploaded a very rough draft of a syslog server written in Ruby. Here’s a link to it. I talked about it in a previous blog post, and I think now its ready to show the world. Here’s a quick breakdown of some of the key features:

• Uses ActiveRecord for DB backend
• Heavily Multithreaded
  • Makes extensive use of Queues and Mutexes
  • Works in Ruby 1.9, JRuby, and Mostly in MacRuby for Native threads. (Obviously) Uses green threads in Ruby 1.8
  • Utilizes Thread Pools to maintain a set number of Threads
• Listens for UDP, TCP, or both.
• Uses a YAML config file
• Fully documented with RDoc
• Performs exceptionally under heavy load. I tested 10,000+ simultaneous connections with the included test client (under /test/).
• Written with MVC in mind. (I say “in mind” because there really is no “view” portion to this app)

That said, its still pretty new, and I’m still working on it. I welcome any comments about the code or any bugs encountered. Keep in mind that this SHOULD NOT BE USED IN PRODUCTION. I offer no warranty, implied or otherwise. I license this code under the GPLv3, even if I forgot to include a copy of the license in the tarball.

All-in-all, its actually a pretty simple app. Not that much actual code involved; just a little cleverness on how to process things one piece at a time. Here’s some of my favorite code, along with some parts that help it make more sense:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

# ...
        # This methods tells the server to pull all logs from the LogQueue and 'process' them.
        # Essentially, this moves logs from the incoming LogQueue to a process ThreadPool for
        # controlled processing. This method prevents an insurge of messages from eating up all
        # the server's CPU by limiting the number of logs processed at a time. After the log is
        # parsed in the process pool, it is queued up for storage in the storage ThreadPool.
        def process_logs
                until @log_queue.empty?
                        @queue_flushed = false if @queue_flushed
                        log = pop_log
                        @process_pool.add_job {
                                puts "Processing Log: #{log.object_id.to_s(16)}"
                                parsed_log = @parser.parse log
                                @storage_pool.add_job {@storage.store_log parsed_log}
                        }
                end
                if !@queue_flushed
                        puts "Log Queue Flush Complete @ #{Time.now}" if !logs_queued?
                        @queue_flushed = true
                end
        end
        
        # Utilizes the LogMutex to synchronize initial storage of log packets in the LogQueue
        def enqueue_log(log_data)
                @mutex.synchronize { @log_queue.push log_data }
        end
        
        # Utilizes the LogMutex to synchronize removing log packets from the LogQueue
        def pop_log
                return @mutex.synchronize { @log_queue.pop }
        end
        
        # Utilizes the LogMutex to synchronize checking how many logs are currently queued in the
        # LogQueue.
        def logs_queued?
                return @mutex.synchronize { @log_queue.size > 0 }
        end
# ...

Hopefully, with more work, this could have some serious potential.

I added some minor improvements to my blog. Some of these feature include:

• All tags now offer feeds, even if they aren’t in the “Popular Tags”. Just click on the tag, and the grey banner contains a feed link.
• All successful searches (i.e., those that return results) allow for subscriptions via Atom feeds. Similar to tag feeds, they exist in the grey banner that details the search term.
• All uploaded media is available and searchable from the newly added “Media” link under the also newly added “Extras” section on the right.
• Several small improvements to admin areas…

Here’s some code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

def feeds
    case params[:feed]
      when "all"
        @entries = Entry.find(
          :all, 
          :limit => 25, 
          :order => "updated_at DESC"
        )
      when "by_tag"
        @entries = Tag.find_by_name(params[:tag]).entries.find(
          :all, 
          :limit => 25, 
          :order => 'updated_at DESC'
        )
      when "by_search"
        query = "%#{params[:q]}%"
        @entries = Entry.find(
          :all, 
          :conditions => ["body LIKE ?", query], 
          :limit => 25, 
          :order => 'updated_at DESC'
        )
    end # case params[:feed]
    respond_to do |format|
      format.atom  { render :layout => nil }
    end
end

I was trying out recursive programming in Ruby the other day. I figured a good way to try it out was using a pretty well known mathematical sequence called the Fibonacci sequence. In math, the sequence is defined as

with seed values of

F₀ = 0 and F₁ = 1

so its a prime candidate for a recursive algorithm in Ruby. I started off with a working, but very slow, method that looks like this:

1
2
3
4
5
6
7
8
9

def fib(n)
  if (n == 0)
    return 0 
  elsif (n == 1)
    return 1
  else
    return fib(n - 2) + fib(n - 1)
  end
end

For the first few numbers in the sequence this method performs well enough, but as the number gets larger, things start slowing way down. For example:

1
2
3
4
5

fib 6  #=> 8
fib 9  #=> 34
fib 20 #=> 6765
fib 30 #=> 832040 (took about 5 seconds to complete)
fib 40 #=> ? (waited for over a minute with CPU pegged...)

So, obviously this stuff is working, but its not really very practical. Next I took a stab at using an array to create an in-memory cache, which sped things up immensely:

1
2
3
4
5
6
7
8
9
10
11
12
13

@@cache = [] # this is an in-memory cache
def fib_cached(n)
  if @@cache[n]
    return @@cache[n] # return the cached value, if it exists
  else
    if (n == 0) || (n == 1)
      @@cache[n] = n
    else
      @@cache[n] = fib_cached(n - 2) + fib_cached(n - 1)
    end
    return @@cache[n] # always put the value in cache, then return it
  end
end

This speed difference was unbelievable:

1
2
3
4
5
6
7

fib_cached 6   #=> 8
fib_cached 9   #=> 34
fib_cached 20  #=> 6765
fib_cached 30  #=> 832040
fib_cached 40  #=> 102334155
fib_cached 99  #=> 218922995834555169026
fib_cached 300 #=> 222232244629420445529739893461909967206666939096499764990979600

I tried different numbers up to 999 and wasn’t able slow it down at all, which is great.

Finally, I found a shortcut on a blog (which I can’t remember the URL for), that uses the Golden Ratio (phi) to very quickly find numbers in the Fibonacci sequence. It works very well, but I think since I’m not really using Ruby’s more advanced math modules, the numbers skew a little (then a lot) as the sequence number gets large.

1
2
3
4
5
6

module Math
  Phi = (Math.sqrt(5) + 1) / 2
end
def fib_by_phi(n)
  (((Math::Phi ** n) - ((1 - Math::Phi) ** n) ) / Math.sqrt(5)).to_i
end

Although this was extremely fast, it didn’t quite produce accurate results at higher sequence numbers:

1
2
3
4
5
6
7

fib_by_phi 6   #=> 8
fib_by_phi 9   #=> 34
fib_by_phi 20  #=> 6765
fib_by_phi 30  #=> 832040
fib_by_phi 40  #=> 102334155
fib_by_phi 99  #=> 218922995834555891712
fib_by_phi 300 #=> 222232244629422676106398124069050176556246085874450435841982464

In the end, I found that the cached recursive algorithm was the best balance of speed and reliability, trading off memory for CPU cycles. Here’s some code that you can use to test things and get a very precise measurement of performance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

def stop_watch(&block)
  start_time = Time.now
  puts "Started at #{start_time}"
  begin
    yield
  rescue Exception => e
    puts "Block died: #{e}"
  end
  end_time = Time.now
  puts "Ended at #{end_time}"
  puts "Finished in #{end_time - start_time} seconds."
end
#run each method in a stop_watch
stop_watch do
  puts fib_cached 9999
end
stop_watch do
  puts fib_by_phi 9999
end

Sometimes you need to work with numbers in Ruby. We all do it, and its necessary. But sometimes you run across a situation where you need to take a string that looks like a number, and convert it to a number. For instance, let say we’ve got the following:

number = "3"

Now lets say that I want to use this in some math… It won’t work very well:

wrong_answer = number * 14 # => TypeError: String can't be coerced into Fixnum

So, how can we make this work? Take a look at this:

answer = Integer(number) * 14 # => 42

This isn’t the only method that provides this kind of functionality. Need to work with Floats? Try this:

1
2

number = "3.14"
answer = Float(number) * (4**2) # => 50.24

Just thought I’d share some useful Ruby code. I haven’t posted code in a while, and I figured it might help somebody out there.

I was messing around at work the other day with our OpenVPN server, and I was kind of tired of how it authenticates against a flat file I was using to store users and passwords (encrypted, ofcourse). So I rewrote the openvpn auth script in Ruby, and I had it hit our LDAP server instead. Wasn’t too incredibly challenging, but I thought I’d share my success, just in case it helps somebody else out there. Here’s a snippet that shows LDAP auth how it works:

1
2
3
4
5
6
7
8
9
10
11
12
13

def check_auth(user, pass)
  require 'ldap'
  ldap_user = "uid=#{user},#{@basedn}"
  begin
    conn = LDAP::Conn.open(@ldap_host, @ldap_port)
    conn.start_tls
    conn.bind(ldap_user, pass)
    return conn.bound?
  rescue Exception => e
    puts "Authentication failed: #{e}" if @verbose
    return false
  end
end

It could probably be improved, but the point is, it does what I need and works pretty well.

In the beginning, my blog didn’t have any way to notify me when someone comments on an entry. After a while, I added an ActionMailer class (called Notifier) to email me when a comment is saved. Realizing that this might open up the door for people to spam me, I added some very basic spam checking, looking for certain inappropriate key words through regular expression matching. Maybe some day I’ll actually try and tie-in Spamassassin or something.

Today, I added a couple more advancements in regards to comments. Now, it also sends me an Instant Message through Jabber (XMPP) notifying me of a new comment. I also decided to move the sending of the email and the instant message to its own thread because I noticed some slowness when testing new comments. Now things are fast and smooth again.

Here’s the code I used to make the IM and email happen:

Add this to the Comment model:

1
2
3
4
5
6
7

...
  protected
  def validate
    SPAM_REGEXES.each do |r|
      errors.add("body", "fails SPAM checks") if body.match(r)
    end
  end

Add this to the Comment controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

...
    if @comment.save
      t = Thread.new do
        Notifier.deliver_comment_notification(@comment.entry.author, @comment)
        begin
          # Connect to the server and set presence to available for chat.
          client = Jabber::Simple.new('jlg_blog_bot@jabber.org/blog', JABBER_PASSWORD.to_s)
          client.deliver(
            @comment.entry.author.email, 
            "You have a new comment on your blog entry titled '#{@comment.entry.title}'",
            :normal
          )
          sleep(1)
          client.disconnect
          logger.info "Sent XMPP message about: '#{@comment.entry.title}'"
        rescue
          logger.warn "Sending of XMPP message failed..."
        end
      end # new thread
    else
      # If the comment wasn't saved, it was probably marked as spam
      flash[:warn] = "Your comment was not saved. Perhaps it was flagged as spam."
    end
...

You still need to modify the ‘config/environment.rb’ file to include definitions for JABBER_PASSWORD and SPAM_REGEXES.

Ever need to use a URL with an underscore in Ruby? Normal URI processing fails because underscores are not allowed in host names of DNS records (except for SRV records). Since I’ve come across situations where its still necessary to parse the invalid URLs, I wrote this class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

class FakeURI
  attr_accessor :host
  attr_accessor :port
  attr_accessor :path

  def initialize(url)
    @scheme = /^([htps]+):/.match(url)[1]
    @userinfo = nil
    @host = /^http(s)?:\/\/([a-zA-Z0-9_.-]+)(:[0-9]+)?(\/)?.*$/.match(url)[2]
    @port = /^http(s)?:\/\/.*:([0-9]+)(\/)?/.match(url)[2]
    @registry = nil
    @path = /^http(s)?:\/\/(.*):([0-9]+)(.*)$/.match(url)[4]
    @opaque = nil
    @query = nil
    @fragment = nil
  end

  def self.parse(url)
    return FakeURI.new(url)
  end
end

Here’s an example of how it can be used:

1
2
3
4
5
6
7

begin
  uri = URI.parse(url) # Pull apart the url into protocol, port, host, and path
  # The above code throws an exception for URIs with an underscore "_" in the host
  # name. Thus, the following rescue statement.
rescue
  uri = FakeURI.parse(url) # If the normal URI object throws an error, try this one
end

Now, assuming you have a useable URL to begin with, you will have the info you’re looking for one way or another. This isn’t perfect, and it obviously doesn’t create a complete URI object, but it worked for my needs.

This is my first entry in my Ruby on Rails weblog. This uses some very neat features that were sorely lacking in pretty much all the blog software I could find. Some of these are:

• Textile support (using RedCloth) for wiki-style syntax
• Coderay syntax highlighting
• Full support for tags
• Full-text search of all entries
• Live preview of entry addition via AJAX
• All pages are, as far I can tell, fully compliant with XHTML 1.0 Strict

That said, I think this is going to be exciting!

Here’s an example of the syntax highlighting, displaying the code that makes syntax highlighting work, no less:

1
2
3
4

# Markup parsing (both coderay and textile)
def unmarkup(text)
  return parse_coderay(textilize(text))
end

Let me know what you think!