Tech-Blog: Category technology

On Bootstrapping Active Directory for EC2's Windows Images

Mike — Wed, 29 Apr 2009 09:09:00 -0400

Amazon's EC2 service now includes a number of Windows Server 2003 images, starting at $0.125 per hour. These Microsoft sanctioned images are a great alternative to the QEMU shenanigans from the past.

One of the advantages of Amazon's EC2 service is the ability to provision virtual machines via scripts, and without human interaction. Here's a description of my approach to bringing an instance of Active Directory online via script.

Prerequisites:

Amazon EC2 account
command line RDP client (e.g. rdesktop)
dcpromo answer file (I found Daniel Petri's article helpful)

Your RDP client will need to support two features: file sharing, and remote script execution. I've found the Mac RDP clients tough to use via script, but Ubuntu's rdesktop works just fine in a VirtualBox VM.

Starting the Windows Server VM

Choose the Basic Microsoft Windows Server 2003 image or the Basic Microsoft Windows Server 2003 with Authentication Services image; a small standard instance type should be sufficient.

(The Firefox elasticfox EC2 plugin makes it easy to manage windows instances.)

Once your instance is online, you'll need the public DNS name and administrator password for the instance.

Installing Active Directory

Normally, AD is installed and configured by running dcpromo and working through the relevant dialog boxes. Unattended installs are supported via the /answer argument:

   C:> dcpromo /answer:%path_to_answer_file%

Running this command on an EC2 instance poses two challenges: how to run a windows command remotely from a script, and how to copy our dcpromo answer file from our local host to EC2. Fortunately RDP solves both problems. Using rdesktop one can define a script to run on session startup, and a local folder to share with the remote host.

Install Active Directory remotely using rdesktop as follows:

   mpierson:$ rdesktop -r disk:share=/home/mpierson/winshare/ \
     -u Administrator -p <admin_password> \
     -s "dcpromo /answer:\\\\tsclient\\share\\dcpromo.install" \
     <windows_hostname>

where /home/mpierson/winshare/dcpromo.install is the local path of the dcpromo answer file for your AD configuration. (The RDP file share defined by the -r argument will appear in the Windows host network share list with a host name of tsclient.)

FWIW, here is an example answer file for dcpromo.

Access Your AD Domain

Use rdesktop to access your new AD domain as follows:

   mpierson:$ rdesktop 
     -u Administrator -p <admin_password> \
     -d <win_domain_name> \
     <windows_hostname>

where win_domain_name is the DomainNetBiosName defined in your dcpromo answer file.

On Bootstrapping Oracle's 10g XE EC2 Image

Mike — Tue, 28 Apr 2009 12:37:00 -0400

Oracle has made a number of their products available for use on Amazon's EC2 service. Unfortunately, each instance of their 10g XE RDBMS image must be configured manually via ssh before use.

Turns out that the 10g XE image hijacks the bash profile script to force manual configuration. If one was motivated to enable unattended configuration of the image, the bash profile script could be un-hijacked with scp:

  mpierson:$ scp -i EC2_KEY my-bash-profile.sh root@<EC2-HOST-NAME>:/root/.bash_profile

BTW, the hijack script overwritten by scp above is also a good starting point for an script that would bring the database online in an unattended scenario. Here's what it does:

force acceptance of Oracle EC2 license
force change to oracle system user
set Oracle environment, including host name
force change to SYSTEM and SYS oracle db accounts

Extracting IdM Configuration from a Running Instance

Mike — Fri, 24 Apr 2009 22:16:00 -0400

An IdM customer recently found themselves in a pickle: they wanted to integrate additional resources into their IdM deployment, but did not have access to the original configuration files or CBE.

Here are the steps we took to extract configuration from their production repository:

export configuration from running system via lh
split export file, one file per top level element
mangle XML objects (remove date based attributes, GUID etc)
repeat above for reference, out-of-the-box, IdM deployment
diff XML object sets (custom deployment vs. reference deployment)

And the details...

Export Configuration

Use the lh command to extract the running configuration:

   mpierson:$ export WSHOME=/var/lib/tomcat/webapps/idm
   mpierson:$ $WSHOME/bin/lh console -c "export /tmp/export.xml"

Note: the export process may take some time, and consume significant cycles on the host and the repo; the resulting export.xml file will be of the order of 100Mb, depending on the number of user/resource accounts.

Split Export File

To facilitate comparison of the custom configs to the reference deployment, split the export file into many files, one per top level waveset element:

   mpierson:$ xsltproc split-waveset.xslt /tmp/export.xml

We used a simple XSL transform that writes each child of the top level waveset element to a file, using file names based on name or id attributes, if available.

Note: the example XSL transform will write the files to a directory called 'split', and there will potentially be many files created.

Mangle XML Objects

Again, to facilitate comparison of the custom config objects to the reference deployment, we mangled the split XML files to remove date based attributes, owner/modifier attributes, and GUIDs:

   mpierson:$ ./remove-transient-attrs.sh split-dir idm-key-prefix

... where split-dir is the name of the directory containing XML files to be cleansed, and idm-key-prefix is the first 5-6 characters of the instance ID prefix (i.e. '123456' in "#ID#123456789ABC...")

Note: here is the bash script that utilized find and perl to strip the relevant attributes.

Repeat for Reference Deployment

Repeat steps 1 thru 3 to produce an reference set of waveset objects. Ensure that you reference IdM deployment matches the version, including hotfixes, of the running instance being analyzed.

Catalog Differences in Custom Deployment

We applied diff recursively over the two sets of waveset objects:

   mpierson:$ diff -N -r --brief -w reference-elements/ custom-elements/

The results of a diff will likely include a number of run-time objects, including User, Account, XmlData, and Syslog elements. Filter run-time objects from the diff for a better view of configuration:

   mpierson:$ diff -N -r --brief -w reference-elements/ custom-elements/ |\
    grep -v Account| grep -vi syslog |\
    grep -v TaskInstance | grep -v "User-" |\
    grep -v TaskResult | grep -v XmlData |\
    grep -v WorkItem

We used a similar approach to produce a list of the XML objects that define the IdM configuration:

   mpierson:$ diff -N -r --brief -w reference-elements/ custom-elements/ |\
    grep -v Account| grep -vi syslog |\
    grep -v TaskInstance | grep -v "User-" |\
    grep -v TaskResult | grep -v XmlData |\
    grep -v WorkItem |\
    awk '{print $4;}' > files.txt

Notes: the results of this process provides a starting point for a rigorous reverse engineering endeavour. Manual inspection of the results, and extensive testing are recommended!

Amazon EC2 Diary I

Mike — Wed, 07 Jan 2009 21:30:00 -0500

I've been an Amazon EC2 evangelist for the past couple of years. The EC2 'cloud computing' service is an early entry in the on-demand hosting market, featuring practically unlimited computing resources (CPUs, storage and bandwidth) provisioned in real time via public APIs.

With a tip of the hat to Tim's Android diary:

I suspect that my experiences are going to be shared by quite a few people in the not-too-distant future, so why not record them?

My experience with EC2 involves deployment of JES environments, typically for development and testing of IAM applications. In future posts I'll share my approach and some concrete examples of how I have leveraged EC2 to save time and money.

For now, a short summary of the EC2 features that have provided value.

Per-Hour Billing

Most of us do not work 24 hrs per day seven days per week, so it makes not to pay for dedicated computing resources when developing applications. The EC2 rates are reasonable relative to commercial hosting services, and I suspect an order of magnitude more economical than buying servers and deploying traditional virtual hosts.

Multi-OS Support

The EC2 service started with Linux OS support only, but has since added Solaris and Windows Server 2003. I use all three.

Scriptable API

There are no online forms, or phone calls, or visits to Best Buy. Provisioning of EC2 resources can be fully automated. Use bash, Java, Perl, Ruby..., whatever.

Community & Tools

There are open source API libraries in your favourite language, commercial management utilities, a Firefox plugin available. And of course there are many pre-built EC2 images available to the public.

iPod video

Mike — Thu, 13 Oct 2005 11:04:00 -0400

The worst kept secret is now public.

Will the video iPod move mainstream video off of cable? Maybe not. But it may be the trigger for mainstream video blogs and pod casts, downloadable TV shows, live video. Something big is going to happen. It will also be interesting to see if/how Sony reacts with their PSP.

There may be an opportunity for a bloglines-like app for notification of new video content -- an RSS based TV guide. Or maybe a Flickr for home video?

Web quickies

mop — Thu, 17 Mar 2005 12:09:00 -0500

Some tidbits from my Bloglines RSS subscriptions.

Phonetags

Via Smart Mobs, Phonetags allows cell phone users to create bookmarks for BBC6 radio content, complete with metadata. A clever way to create playlists for users ’on the go’, and a taste of what is to come as radio and TV merge with Internet enabled mobile devices.

Spell with Flickr

Flickr, one of the leading online photo organizers, opened their API some time ago, allowing third party applications to flourish. Spelling with images is a little cheezy, but it’s a start (note the smart URI for the In-Touch example). Amazon, one of the first to open their API, can do something similar with conver art. Via Smart Mobs.

Yahoo Mobile RSS

Yahoo has added RSS aggregation to it’s list of mobile offerings. The UI seems a little primitive, although it’s tough to comment much without seeing the aggregator in action. See also the Yahoo news article. I’d like to hear comments from anyone who uses Yahoo on their mobile. Via MMOM.

Foray into 64-bit computing

mop — Thu, 12 Aug 2004 11:00:00 -0400

A summary of my experience to date with AMD’s 64-bit platform...

AMD has been developing 64-bit processors since 1999 or so, and released their first commercial chips in 2003. Large word systems have been around for some time (SPARC, MIPS, etc.), so why now and why AMD? Much of the recent interest in 64-bit systems is due to the scaling of commodity hardware - processors are fast and memory is cheap. In theory, fast hardware will diminish the overhead associated with larger data chunks and registers, and 64-bit systems can utilize large amounts of RAM. The new AMD processors are interesting because they are backward compatible with Intel 32-bit instruction sets (i.e. you can use binaries compiled for x86) and because the chips are priced to compete with 32-bit commodity hardware.

32-bit emulation

Is it marketing hype or is AMD64’s backwards compatibility a stroke of genius. Perhaps on a desktop, where you’re running a mix of compiled and commercial software it’ll be nice to have a choice. The reality is probably not that simple, but I suspect OSs will adapt. On servers, I suspect that most 64-bit systems will run in a pure 64-bit environment.

larger address space

The AMD address space is larger than conventional systems (40-bit, I think) allowing many gigabytes of RAM (terrabytes?) to be addresses without funky munging that is required in 32-bit systems. The 4GB address space of 32-bit systems is becoming a practical limit in today’s commodity systems. This, I think, is AMD64’s current sweet spot.

NetGear router upgrade

mop — Fri, 30 Jul 2004 21:53:00 -0400

If anyone else is having a hard time ugrading the firmware of their NetGear MR814 wireless router using Mozilla/Firefox, then Marty might have the answer. The trick is to include the Upgrade=" Upload " parameter.