Apr 12, 2013

Chef and Crowbar for setting up OpenStack clouds - with openSUSE and SUSE Cloud

To easily setup an OpenStack-based cloud on server hardware, we at SUSE use the open source Crowbar project. Crowbar eases the setup of all the different OpenStack components and thus reduces the time spent deploying and managing an OpenStack-based cloud.

Crowbar is a platform for server provisioning and deployment from bare metal. It provides server discovery, firmware upgrades, and operating system installation using PXE Boot, and it deploys applications on top of functioning operating systems using Chef.

Currently development is happening on Crowbar 2 and I'd like to give a short overview of a few things that are done here to make this work useable for openSUSE and SUSE Cloud - SUSE's OpenStack based cloud offering.

Moving to a common Chef cookbase base

So far, Crowbar comes with its own Chef cookbooks to setup OpenStack, a forked and heavily adapted version.

We'd like to have one common set of Chef cookbooks that are shared between Crowbar and administrators using Chef directly. As Rob Hirschfeld said: "The real victory comes when multiple deployments use the same trunk instead of forking".
The additional benefit here is that these cookbooks will then also work without Crowbar on e.g. openSUSE and SUSE Linux Enterprise.

Right now it looks like the awesome cookbooks done by AT&T and Rackspace (e.g. https://github.com/att-cloud/cookbook-keystone/blob/master/README.md) are the best option for going forward. But these cookbooks have two limitations for our usage: they do not handle openSUSE and SUSE Linux Enterprise and are not ported for Grizzly yet. So, this gives us the next tasks:

Enhance existing Chef cookbooks to know about openSUSE and SUSE - and for Grizzly

We're enhancing these cookbooks now so that they support OpenStack Grizzly and we're also adding support for servers running openSUSE and SUSE Linux Enterprise.

Once the cookbooks are usuable on their own, they can be integrated into Crowbar.

Integrate the Chef cookbooks into Crowbar

To use the cookbooks from Crowbar, Crowbar needs to pass parameters to
them. This concept is called "Attribute Injection" and Judd Maltin has explained this in detail in a great blog post.

Crowbar 2 and Chef / Puppet

Of interest to some might be that in the coming Crowbar 2, the Chef usage has been hidden behind an extra layer called "Jig". Jig was introduced to overcome some design limitations of Crowbar, the extra abstraction also gives a more flexible framework. Now other configuration management options can be added besides Chef. Puppet is one framework that the Crowbar team would like to support as backend and this work is currently postponed after the initial Crowbar 2 release.

Apr 9, 2013

OpenStack Grizzly Packages for openSUSE and SUSE Linux Enterprise

Last week the OpenStack project released its 7th release named Grizzly and the packages are now available for openSUSE and SUSE Linux Enterprise via the openSUSE Build Service. Packages can be downloaded from the download server.

The repositories contain all the core OpenStack packages plus incubated projects that will be part of core in the Havana release (Heat for VM orchestration and Ceilometer for metering). Additional package dependencies are included as well.

For discussion about these packages, use the opensuse-cloud at opensuse.org mailing list and the freenode channel #opensuse-cloud. To learn more about OpenStack on openSUSE, see also our wiki page.

Mar 11, 2013

Moving to open development: OpenStack / Crowbar / Chef on SUSE

This blog post was written by Adam Spiers and Andreas Jaeger.
Here at SUSE we have based our business on Free and Open Source Software for over 20 years, so it is nothing new to say that we strongly believe in open development and collaboration as a way of producing software of the highest quality.
Therefore we are pleased to announce that we have opened up our work on development and packaging of OpenStack, Crowbar, and Chef, for both openSUSE and SUSE Linux Enterprise. This work is happening publically in the Open Build Service, and we warmly invite everyone to use the results and also join in with development, testing, documentation and packaging.
Open development is a process that does not end with code on public servers, but involves ongoing open communication as well. Please speak up on the opensuse-cloud mailing list if you see any areas where we could improve further!

OpenStack

OpenStack is our preferred cloud management platform, due to its open nature and backing from an already huge and rapidly growing community. We have packaged both OpenStack's Folsom release and the beta packages for the upcoming Grizzly release and have made them available in the Open Build Service.

Crowbar

OpenStack's incredible flexibility can also make it difficult to deploy. Therefore to ease the set-up of all the different OpenStack components out of the box, we are supporting the open source Crowbar project.  Our partners and customers find that using Crowbar dramatically reduces the time spent deploying and managing an OpenStack-based cloud.
Crowbar is a platform for server provisioning and deployment from bare metal. It provides server discovery, firmware upgrades, and operating system installation using PXE Boot, and it deploys applications on top of functioning operating systems using Chef.

Opening up Crowbar development

Similarly to how OpenStack was born at RackSpace and later grew into an independent entity in the form of the OpenStack Foundation, Crowbar was originally developed by Dell engineers, and is now a fully-fledged Open Source project involving close collaboration with SUSE and others. There are weekly public meetings, a public mailing list, a #crowbar IRC channel, public Trello boards and so on. As an indication of the project's independent nature, the authoritative location for the git repositories has changed from https://github.com/dellcloudedge to https://github.com/crowbar, and a new homepage is currently under construction.

Crowbar 2.0

Work on the upcoming 2.0 release of Crowbar is proceeding at a furious pace, and packages for openSUSE and SUSE Linux Enterprise are available from the OBS. You are very welcome to join in and find out more!

Chef packages

Chef is an Open Source configuration management tool that allows remote administration of systems at scale. It has a thriving community which has a large overlap with the OpenStack community. We have packaged both Chef 10 and Chef 11 for openSUSE and SUSE Linux Enterprise.

Tying it all together

Since each major release of these projects is packaged separately in the Open Build Service, you are free to "mix'n'match" which of these components you want to use together:
  • OpenStack Essex
  • OpenStack Folsom
  • OpenStack Grizzly
  • Crowbar 2.0
  • Chef 10
  • Chef 11
Of course, each combination will work differently. To try any of them out, simply navigate to one of the project's Repository tab to obtain the link to the relevant download repository. Then you can add that to your host's list of software repositories in the normal way via YaST2 or zypper, and start installing packages from it. Alternatively, you can install directly via one-click install.

Automation and Continuous Integration

We use Jenkins to automatically package changes from upstream git repositories into rpms within an openSUSE environment, and if they successfully build and install, Jenkins commits those changes to the Open Build Service (OBS) which then automatically builds and publishes packages and images. Jenkins also does unit tests as well as full stack tests. By automating the packaging and testing process, we can spend more time on development while maintaining high quality, and gain more clarity around where problems exist.
You can also read in more detail about our development model using the Open Build Service.

Finding out more

If you have questions, ask them on the opensuse-cloud mailing list or on the recently launched #opensuse-cloud freenode IRC channel.
You can also find more details via the openSUSE wiki: This is work in progress, so feel free to help improve anything!

Feb 25, 2013

Coordinating efforts to create OpenStack packages for openSUSE and SUSE Linux Enterprise Server

Engineers from B1 Systems and SUSE have now joined forces to package OpenStack in the Open Build Service for openSUSE and SUSE Linux Enterprise. The B1 Systems engineers bring in their OpenStack consulting and packaging expertise, the SUSE team their packaging and hardening expertise from building a commercial OpenStack based release with SUSE Cloud.

The work is done in the Open Build Service which allows building of packages for various distributions and versions of it. There are stable packages available from the latest OpenStack release, Folsom, and beta packages of the upcoming Grizzly release. These OpenStack packages are build now for openSUSE 12.2, openSUSE 12.3 and SUSE Linux Enterprise 11 SP2.
The joint development project is contained in the Cloud:OpenStack and its subprojects.

Additionally, OpenStack Folsom is part of the upcoming openSUSE 12.3 release. Thus, you can build, administrate and use an OpenStack cloud within openSUSE.

If you like to use OpenStack packages or join the team, please join us on the mailing list opensuse-cloud@opensuse.org (subscribing information is here).

I'll talk in future posts on the development progress of these packages in the Open Build Service. In the meantime, to get more information about OpenStack on openSUSE, see the openSUSE OpenStack Portal.

Some background: B1 Systems is a consulting partner of SUSE and offers support for OpenStack on openSUSE and SUSE Linux Enterprise.  Some of their engineers have been packaging OpenStack for over two years now (starting with Bexar) and made those packages available through the Open Build Service for both openSUSE and SUSE Linux Enterprise. SUSE has delivered last year its SUSE Cloud 1.0 release that is based on OpenStack Essex.

Dec 21, 2012

Easy way to install TeX packages for openSUSE 12.3

With the upcoming openSUSE 12.3, the TeX installation is much more modular - but contains now all of TeXlive. Werner added a nice trick to install packages.
If you get an error like:
LaTeX Error: File `multirow.sty' not found.
You can now easily install the missing package with just saying "zypper in 'tex(multirow.sty)'".
Similar, if a font file is missing, e.g. you get a message like this:

! Font U/pzd/m/n/10=pzdr at 10.0pt not loadable: Metric (TFM) file not found.you can install the file with "zypper in 'tex(pzdr.tfm)'".


So, with the new capability 'tex', you can easily install missing files.

Nov 20, 2012

Glibc 2.17 on the finishing line

In the next few days David Miller as GNU C Library (glibc) 2.17 release manager will call a freeze for glibc development and I've packaged glibc for openSUSE and compiled the whole distribution with it - and it looks fine.
For those curious, what's in the new glibc, here's an incomplete list of the major changes. For the full list, see ChangeLog file and the NEWS file. The list comes from the current NEWS file and I've only copied entries relevant to openSUSE:
  • Port to ARM AArch64 contributed by Linaro.
  • The new function secure_getenv allows secure access to the environment, returning NULL if running in a SUID/SGID process.  This function replaces  the internal function __secure_getenv.
  • SystemTap static probes have been added into the dynamic linker. Implemented by Gary Benson.
  • Optimizations of string functions strstr, strcasestr and memmem. Implemented by Maxim Kuvyrkov.
  • The ttyname and ttyname_r functions on Linux now fall back to searching for the tty file descriptor in /dev/pts or /dev if /proc is not available.  This allows creation of chroots without the procfs mounted on /proc.
  • The `crypt' function now fails if passed salt bytes that violate the  specification for those values.  On Linux, the `crypt' function will consult /proc/sys/crypto/fips_enabled to determine if "FIPS mode" is enabled, and fail on encrypted strings using the MD5 or DES algorithm when the mode is enabled.
  • The `clock_*' suite of functions (declared in <time.h>) is now available directly in the main C library.  Previously it was necessary to link with -lrt to use these functions.  This change has the effect that a single-threaded program that uses a function such as `clock_gettime' (and is not linked with -lrt) will no longer implicitly load the pthreads library at runtime and so will not suffer the overheads associated with multi-thread support in other code such as the C++ runtime library.
Also, 125 bugreports have been marked as fixed, a couple of changes from eglibc got merged (especially for cross-testing) and some cleanups of the code were done.

With the current git development version packaged as rpm, the openSUSE Build Service compiled the whole distribution on x86-64. There were three different kind of problems in packages which showed up as build errors:
  1. Some packages had missing includes (e.g. signal.h or stdint.h). Those are easily fixed by including the header defining the missing types.
  2. The functions setfsgid() and setfsuid() produce warnings when the return value is not checked and thus fail to when -Werror is used. The proper fix is to check whether these functions have a return value less than 0.
  3. clock_gettime() was moved from librt to libc. Some configure scripts check only whether clock_gettime is in librt and assume that other functions like mq_gettattr() are in the same library. So, the configure check for clock_gettime() in librt needs to be extended to look for other functions as well.
I did not noticed directly any problems in glibc itself, just these three kind of problems in packages - and had to fix around 10 packages at all.
This was all done with the git version from the 9th of November, I've updated now to the current git version and will retest.
My plan is to push the current glibc package soon to the openSUSE development tree called openSUSE Factory.

Nov 7, 2012

Updates to openSUSE Packaging Guidelines

On the opensuse-packaging mailing list, we've recently formed a team that will take care of the packaging guidelines and introduced a small process to change them.

As part of that process, we're announcing regularly the changes to the packaging guidelines. Since this is a first such announcement, it is not a complete change but just points out a few things from the past few months. In the future, we will send out this email once a month.

The Packaging guidelines can be found in the openSUSE wiki at http://en.opensuse.org/openSUSE:Packaging_guidelines.


The list of changes that I'd like to mention are:
  • New Lua Guidelines
  • Reworked font guidelines
  • Documenting changes in packages
  • Teams involved

New Lua Guidelines

We now have guidelines for lua at http://en.opensuse.org/openSUSE:Packaging_Lua.

Reworked font guidelines

The packaging of fonts has been completely changed, and is documented at http://en.opensuse.org/openSUSE:Packaging_Fonts.

Documenting changes in packages

The openSUSE review team is now also enforcing proper documenting
changes in packages:

First, the .changes entry (rpm changelog) surves two purposes:
  • News for the user
  • History tracking of packaging changes (often referenced in bugs to verify if a user has the latest packaging bugfixes).

Information about updates

A simple "Update to version x.y.z" is, as before, not accepted. There should be some buzz around the update for the user; some major reasons to the upgrade should be listed.
 
Changes on the package itself should be mentioned in a way that any other contributor to the same package can follow traces of why something is the way it is. Commonly, added (build)dependencies are interesting to be seen, special hacks to make something work in a particular way [..]: Always consider that package maintenance is a distributed task and various contributors need to be able to step up at will.

Documenting patch life cycle

The rules about patches are listed at http://en.opensuse.org/openSUSE:Packaging_Patches_guidelines .

Most prominent is likely the mentioning of the patches life cycle, which forces you to mention additions and removals of patches in the changelog. As history shows, this can be helpful if a patch got removed, and later a regression is reported; finding out when a patch was removed can be crucial in reconstructing feature sets (including contacting the contributor that dropped it.. which is easily extracted from the .changes if listed).

The main appeal is to the devel project maintainers / reviewers, to keep out for those rules, to live according to them, as it is frustrating for everybody if a package needs to be declined by the openSUSE Factory Review team:

The dev prj maintainer is the one getting the 'decline' (as it was usually a forwarded request), which often leaves the 'fixing' to the devel project maintainers, where the 'originator' of the fix would have been willing to actually do that...
Note: The review team is not enforcing the usage of patch markup unless the package already follows this convention.

Teams involved, contact

I mentioned two teams previously, these are the openSUSE review team
and the team taking care of the guidelines. You can reach both via the opensuse-packaging mailing list.