Archive for the 'Ruby' Category

« Previous Entries

Realtime scp output with ruby

September 7th, 2012 by pyrat

I have had an annoying problem recently with a shell script where it would not show the output of an scp command in realtime. Initially I was fighting with different methods of capturing output in realtime from a ruby script.

The solution to this which I went for is using IO.popen

eg.

  def stdout_redirect(command)
    f = IO.popen(command)
    while line = f.gets
      puts line
    end
    f.close
  end

However, this was only part of the solution. It wasnt working so I even tried implementing a solution in python.

  import subprocess
 
  def myrun(cmd):
      """from http://blog.kagesenshi.org/2008/02/teeing-python-subprocesspopen-output.html
      """
 
      print "Running " + cmd
 
      p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
      stdout = []
      while True:
          line = p.stdout.readline()
          stdout.append(line)
          print line,
          if line == '' and p.poll() != None:
              break
      return ''.join(stdout)
 
  myrun('scp -r "remote_server:/directory/test_copy" ~/test_files/')

To my surprise, all was quiet so had a check in the terminal and it would output fine. Hmmm.

  file1.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file2.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file3.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file4.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file5.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file6.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00

After some time with mr google, I managed to track down the issue with scp itself. It runs the isatty() function to check and see if the command is being run in a shell. If it is not, then all is quiet! This means ruby, python et al do not get scp output.

The solution thanks to stack exchange was to send all output to a shell!

eg.

scp -r "remote_server:/directory/test_copy" ~/test_files/ > /dev/tty

Hope this helps you if you are having a similar issue!

Posted in Ruby | No Comments »

Install ruby 1.9.3-p125 from source

March 2nd, 2012 by pyrat

On ubuntu server 10.04 you might want to avoid rvm and rbenv and just go straight for installing ruby on your server.

Install openssl development libraries and prerequisites.

sudo apt-get -y install build-essential libssl-dev libreadline5-dev zlib1g-dev

Install yaml

#!/bin/bash
 
cd /usr/src
sudo wget http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
sudo tar xvzf yaml-0.1.4.tar.gz
cd yaml-0.1.4
sudo ./configure --prefix=/usr/local
sudo make 
sudo make install

Install ruby

#!/bin/bash
 
cd /usr/src
sudo wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p125.tar.gz
sudo tar xvzf ruby-1.9.3-p125.tar.gz
cd ruby-1.9.3-p125
sudo ./configure --prefix=/usr/local --enable-shared --disable-install-doc --with-opt-dir=/usr/local/lib --with-openssl-dir=/usr --with-readline-dir=/usr --with-zlib-dir=/usr
sudo make
sudo make install

Install rubygems

  !#/bin/bash
 
  cd /usr/src 
  sudo wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.17.tgz
  sudo tar xvzf rubygems-1.8.17.tgz
  cd rubygems-1.8.17
  sudo ruby setup.rb

This is applicable if you want to run a certain version of ruby globally on your production server. If there is alreadt a version of ruby installed, check /usr/bin/ruby, /usr/bin/gem and /usr/bin/bundle and update the links accordingly.

eg.

  sudo rm /usr/bin/ruby; sudo ln -s /usr/local/bin/ruby /usr/bin/ruby
  sudo rm /usr/bin/gem; sudo ln -s /usr/local/bin/gem /usr/bin/gem
  sudo gem install bundler --pre
  sudo rm /usr/bin/bundle; sudo ln -s /usr/local/bin/bundle /usr/bin/bundle

Happy compiling!

Posted in Ruby | 4 Comments »

Upgrade utf8 content from a latin1 store to a UTF8 store

March 2nd, 2012 by pyrat

When upgrading to ruby 1.9.3 I noticed that the behaviour of interpreting characters from mysql tables has changes. This is due to the character encoding behaviour differing from 1.8.7.

The symptoms being strange characters appear in your content fetched from the db.

Often in ubuntu mysql server installs the default is latin1 when rails stored everything in utf8. In some cases (like mine) you end up having utf8 data stored in a latin1 table.

Dirty conversion script follows.

  DBFROM=site_production
  DBTO=new_site_production
  LOGIN=site
  PASS=XXX
  mysqldump  --extended-insert=FALSE --default-character-set=latin1  -u $LOGIN -p$PASS $DBFROM >site.sql
  cat site.sql |sed  -e 's/DEFAULT CHARSET=latin1;/DEFAULT CHARSET=utf8 COLLATE utf8_bin;/'>site2.sql
  cat site2.sql |sed  -e 's/SET NAMES latin1/SET NAMES utf8/'>site3.sql
  echo " drop database $DBTO; create database $DBTO character set utf8 collate utf8_bin;"|mysql -u $LOGIN -p$PASS
  mysql -u $LOGIN -p$PASS $DBTO <site3.sql

Posted in Rails, Ruby | No Comments »

Ruby 1.9 Tips

February 7th, 2011 by pyrat

A runnable script to show off a lot of ruby 1.9 features. Hopefully this will increase the takeup of 1.9 as it is now stable enough to use for real in production.

Courtesy of igrigorik.

Posted in Ruby | No Comments »

Currency Rates Library

January 26th, 2011 by pyrat

The Rub

I needed a way of grabbing up to date currency information and an easy way to convert currencies using these up to date rates.

Normally this information requires signing up for premium accounts or writing web scrapers to do this for you.

It turns out that the European bank provides a daily xml for free that contains this information. It is specific to the euro but is fairly trivial to convert this into conversion rates for any of the other currency as the base currency.

Currency Rates is born

gem install currency_rates

It is a simple class for getting up to date currency rates in an array. You pass a base currency which the rates will be calculated against. Connects to the european bank to get a daily breakdown of the rates.

Deliberately lightweight to allow you to build on top of.

Example

#!/usr/bin/env ruby
 
require 'rubygems'
require 'currency_rates'
 
parser = CurrencyRates::Parser.new('GBP')
puts parser.engage.inspect

Valid values for the initialize call are:

"USD"
"JPY"
"BGN"
"CZK"
"DKK"
"EEK"
"GBP"
"HUF"
"LTL"
"LVL"
"PLN"
"RON"
"SEK"
"CHF"
"NOK"
"HRK"
"RUB"
"TRY"
"AUD"
"BRL"
"CAD"
"CNY"
"HKD"
"IDR"
"INR"
"KRW"
"MXN"
"MYR"
"NZD"
"PHP"
"SGD"
"THB"
"ZAR"
"EUR"

Example result array:

[{"currency"=>"USD", "rate"=>"1.58482"}, {"currency"=>"JPY", "rate"=>"130.28671"}, {"currency"=>"BGN", "rate"=>"2.26562"}, {"currency"=>"CZK", "rate"=>"28.05560"}, {"currency"=>"DKK", "rate"=>"8.63284"}, {"currency"=>"ILS", "rate"=>"5.71573"}, {"currency"=>"GBP", "rate"=>"1.00000"}, {"currency"=>"HUF", "rate"=>"317.67159"}, {"currency"=>"LTL", "rate"=>"3.99977"}, {"currency"=>"LVL", "rate"=>"0.81552"}, {"currency"=>"PLN", "rate"=>"4.48653"}, {"currency"=>"RON", "rate"=>"4.94642"}, {"currency"=>"SEK", "rate"=>"10.26180"}, {"currency"=>"CHF", "rate"=>"1.49690"}, {"currency"=>"NOK", "rate"=>"9.11266"}, {"currency"=>"HRK", "rate"=>"8.58558"}, {"currency"=>"RUB", "rate"=>"47.14741"}, {"currency"=>"TRY", "rate"=>"2.50090"}, {"currency"=>"AUD", "rate"=>"1.58992"}, {"currency"=>"BRL", "rate"=>"2.64361"}, {"currency"=>"CAD", "rate"=>"1.57892"}, {"currency"=>"CNY", "rate"=>"10.43290"}, {"currency"=>"HKD", "rate"=>"12.33837"}, {"currency"=>"IDR", "rate"=>"14323.90385"}, {"currency"=>"INR", "rate"=>"72.43649"}, {"currency"=>"KRW", "rate"=>"1770.37938"}, {"currency"=>"MXN", "rate"=>"19.10177"}, {"currency"=>"MYR", "rate"=>"4.83614"}, {"currency"=>"NZD", "rate"=>"2.06591"}, {"currency"=>"PHP", "rate"=>"70.20678"}, {"currency"=>"SGD", "rate"=>"2.02989"}, {"currency"=>"THB", "rate"=>"48.82247"}, {"currency"=>"ZAR", "rate"=>"11.25607"}, {"currency"=>"EUR", "rate"=>"1.15841"}]

Code is also available on github

Posted in Ruby | No Comments »

Google Storage Interoperability With Amazon S3

June 27th, 2010 by pyrat


A google server

Google storage was recently released as an open beta for developers to work with. All in all it seems like an Amazon S3 clone and whilst they have their REST api for integration they also support the Amazon S3 api.

Originally I was planning on writing a ruby wrapper of this api but decided to see if I could modify with existing aws-s3 gem to work with google storage.

After some in depth reading of both apis and understanding of the aws-s3 gem I have modified it to also work with google storage.

My github fork contains these modifications.

To use:

 
  require 'aws/s3'
  include AWS::S3
 
  Base.establish_connection!(
      :access_key_id     => 'abc',
      :secret_access_key => '123',
      :default_host => "commondatastorage.googleapis.com"
    )

Note the addition of the default_host to connect to google instead of AWS.

From then on you can use the gem as normal as is described in the Readme.

Posted in Api, Ruby | 2 Comments »

Deploy when github goes down

November 27th, 2008 by pyrat


You are not screwed

As git is distributed, the repository on your local machine is good enough to deploy from.

First start a git server on your local machine.

  git daemon --base-path=/projects/rails_apps/ --export-all

Then change your capistrano recipe to deploy with the copy command and change the repo to point to your local machine.

  set :deploy_via, :copy
  set :repository 'git://127.0.0.1/proj_name'

Where in this example the code resides at */projects/rails_apps/proj_name*

Now capistrano will deploy from your local repository, thus avoiding the currently melted github

There are other options which have been listed by chris wanstrath in the following article

Posted in Rails, Ruby | 1 Comment »

Google Sitemap Generator

February 26th, 2008 by pyrat

maps are good

Google sitemaps are nice for telling google what is where. Often clients want it for SEO or you have a site which has new content all the time and you want to keep google up to date.

Whatever the reason is thats you are interested in these little xml files, the following code allows you to generate a sitemap for a dynamic site in ruby.

Firstly the class:

  require 'net/http'
  require 'uri'
 
  # A class specific to the application which generates a google sitemap from
  # the contents of the database.
  # Author: Alastair Brunton
  class GoogleSitemapGenerator
 
    def initialize(base_url, sources)
      @base_url = base_url
      @sources = sources 
    end
 
    # The main generator method which in turn adds to the path_array from the different
    # sources.
    # Sources are: pages, events, properties
    def generate
      path_ar = Array.new
      @sources.each do |source|
        # initialize the class and call the get_paths method on it.
        path_ar = path_ar + eval("#{source}.get_paths")
      end
      xml = generate_xml(path_ar)
      save_file(xml)
      update_google
    end
 
    # This creates the xml document.
  	def generate_xml(path_ar)
  		xml_str = ""
  		xml = Builder::XmlMarkup.new(:target => xml_str)
 
  		xml.instruct!
  			xml.urlset(:xmlns=>'http://www.google.com/schemas/sitemap/0.84') {
    			path_ar.each do |path|
      	    xml.url {
        	    	xml.loc(@base_url + path[:url])
        			xml.lastmod(path[:last_mod])
        			xml.changefreq('weekly')
     			 }
    			end
  			}	
  		xml_str
  	end
 
  	# Saves the xml file to disc. This could also be used to ping the webmaster tools
  	def save_file(xml)
  		File.open(RAILS_ROOT + '/public/sitemap.xml', "w+") do |f|
  			f.write(xml)	
  		end		
  	end
 
  	# Notify google of the new sitemap
  	def update_google
  	    sitemap_uri = @base_url + '/sitemap.xml'
  	    escaped_sitemap_uri = URI.escape(sitemap_uri)
  	    Net::HTTP.get('www.google.com',
  	                  '/webmasters/sitemaps/ping?sitemap=' +
  	                  escaped_sitemap_uri)
  	end
 
 
  end

You will notice that an array of strings are passed when calling the generator. These are names of object which implement the get_paths method. An example get_paths class method is as follows:

  # for the google sitemap
   def self.get_paths
     path_ar = Array.new
     Property.live_properties.each do |property|
       path_ar << {:url => "/property/#{property.to_param}", :last_mod => property.updated_at.strftime('%Y-%m-%d')}
     end
     path_ar
   end

Basically, you need an array of hashes which each contain the url and the last_mod.

To call this little beastie it is best done from a cron on the production server. An example rake task to do this is as follows:

  namespace :google_sitemap do
    desc "Generate a google sitemap from the site."
    task(:generate => :environment) do
      sources = ['Page', 'Event', 'Property']
      sitemap = GoogleSitemapGenerator.new('http://www.your_url.com', sources)
      sitemap.generate
    end
  end

Remember when you are calling it from a cron to pass the RAILS_ENV. This generator does rely on rails but you could convert it to only rely on ruby by modifying the rake task and changing the RAILS_ROOT reference in the save_file method. Probably can be made to work with Merb but I am unsure of how merb and rake work together. Will hopefully get my hands dirty with Merb sometime soon.

   cd /var/www/apps/site/current /usr/bin/rake RAILS_ENV=production google_sitemap:generate

Posted in Rails, Ruby | 4 Comments »

Throw your own exceptions

February 24th, 2008 by pyrat

exceptions

Throwing and catching exceptions can be a good design pattern in your rails app. Especially when you want to be able to deal with the unexpected in a clean way. Overuse is a no no as with most techniques but it is nice here and there.

I find that a good idea is to make your own exceptions when writing a rails app. This means that standard low level exceptions which often should not be caught are not.

eg.

(in config/initializers/custom_config.rb)

  class ApplicationError < RuntimeError
 
  end

This way in your application code you can do something like the following

  begin
    @user = User.find_by_password_reset_code(params[:id])
    raise ApplicationError if @user.nil?
 
  rescue ApplicationError => msg
    flash[:message] = "Sorry - That is an invalid password reset code. Please check your code and try again. (Perhaps your email client inserted a carriage return?"
    redirect_to logins_url
  end

In other news I have started testing deploying with git and capistrano. It is blazingly fast even for a full checkout. I cant get :remote_cache deploy via working due to an ancient version of git on LTS 6.06. But event with full checkout this is faster than deploy_via :remote_cache with subversion! (This is with 2 slices on a local network.)

Posted in Rails, Ruby | 1 Comment »

Hpricot 0.6 Caveat Setting Attributes

February 6th, 2008 by pyrat

Watch out with Hpricot 0.6 it doesnt seem to work properly with setting attributes. Use 0.5 instead.

sudo gem install hpricot -v0.5

Posted in Ruby | 1 Comment »

« Previous Entries