Additional info on Google maintenance issue

[kenetrix]

Hi All!

Just wanted to give you all a "heads up" on the issues with Google these last few days.

Aside from any "other" planned maintenance by Google during this period, you should also know that Google is in the process of "transitioning" from their use of XML to a Google-created "interface description language" (IDL) and binary format for data interchange, that it is sharing as an open-source project.

Google made code ( protobuf - Google Code ) and documentation ( Developer Guide - Protocol Buffers ) for its "Protocol Buffers" format available on July 7, under the Apache 2.0 open-source license.

Google is using Protocol Buffers for communication among its internal systems that use Remote Procedure Calls systems and accessing persistent data storage. Developers can define data structures using its definition language, and then compile them, producing classes that represent those structures in a particluar programming language, according to a post by a Google engineer
( Google Open Source Blog: Protocol Buffers: Google's Data Interchange Format ), on Google's software engineering team blog.

The classes, which are designated as .proto files, can read and write structured data from data streams. At this time, language support is limited to C++, Java and Python.

The post also noted that Protocol Buffers is an IDL that was designed for simplicity. IDLs are used to create a communications bridge between programs written in different languages, or that reside on different operating systems.

The framework compiles the IDL code - an intermediary format - into a binary format, which is used as a transmission protocol.

Google's documentation claims that structured data can be represented in a third to a tenth of the space required, and transmission times cut to a 20th or even a 100th of the time, compared to XML.

(My initial testing of Protocol Buffers vs. XML have shown these estimates to
be true, and not just hype. Protocol Buffers is far and away a speedier and more efficient method than XML.)

As an example, look a this:

Whereas with XML, you would have to do something like this:

    cout << "Name: "
         << person.getElementsByTagName("name")->item(0)->innerText()
         << endl;
    cout << "E-mail: "
         << person.getElementsByTagName("email")->item(0)->innerText()
         << endl;

....manipulating a Protocol Buffer require fewer lines of code, like so:

    cout << "Name: " <<person.name() << endl;
    cout << "E-mail: " <<person.email() << endl;

Simple example, true. But, apply that efficiency to the thousands and thousands of data exchanges every second of every minute of every day, and you can see the huge impact of this.

I know it's crappy timing, from the 30DC perspective, but it will be hugely beneficial to all of us.

Hope that sheds some additional "light" on the Google outage issue for you all.

-Ken

[AndyWiltshireNZ]

You could have written anything at the end of that post and I would have believed it, hehe, very informative though, thanks

[Caro]

WOW Ken!!!

You know your stuff. And I would love to have that 'dumbed down' into simple language as to what it means.

I am gathering you are saying it will be a little quicker to get your search results?

In amongst that - are you also saying that it will enable us to access it for more searches without getting a slap on the wrist for doing too many?

I am impressed - very. Just not 100% clear.

Caro

[MizzCindy]

Originally Posted by kenetrix

Hi All!

Just wanted to give you all a "heads up" on the issues with Google these last few days.

Aside from any "other" planned maintenance by Google during this period, you should also know that Google is in the process of "transitioning" from their use of XML to a Google-created "interface description language" (IDL) and binary format for data interchange, that it is sharing as an open-source project.

Google made code ( protobuf - Google Code ) and documentation ( Developer Guide - Protocol Buffers ) for its "Protocol Buffers" format available on July 7, under the Apache 2.0 open-source license.

Google is using Protocol Buffers for communication among its internal systems that use Remote Procedure Calls systems and accessing persistent data storage. Developers can define data structures using its definition language, and then compile them, producing classes that represent those structures in a particluar programming language, according to a post by a Google engineer
( Google Open Source Blog: Protocol Buffers: Google's Data Interchange Format ), on Google's software engineering team blog.

The classes, which are designated as .proto files, can read and write structured data from data streams. At this time, language support is limited to C++, Java and Python.

The post also noted that Protocol Buffers is an IDL that was designed for simplicity. IDLs are used to create a communications bridge between programs written in different languages, or that reside on different operating systems.

The framework compiles the IDL code - an intermediary format - into a binary format, which is used as a transmission protocol.

Google's documentation claims that structured data can be represented in a third to a tenth of the space required, and transmission times cut to a 20th or even a 100th of the time, compared to XML.

(My initial testing of Protocol Buffers vs. XML have shown these estimates to
be true, and not just hype. Protocol Buffers is far and away a speedier and more efficient method than XML.)

As an example, look a this:

Whereas with XML, you would have to do something like this:

    cout << "Name: "
         << person.getElementsByTagName("name")->item(0)->innerText()
         << endl;
    cout << "E-mail: "
         << person.getElementsByTagName("email")->item(0)->innerText()
         << endl;

....manipulating a Protocol Buffer require fewer lines of code, like so:

    cout << "Name: " <<person.name() << endl;
    cout << "E-mail: " <<person.email() << endl;

Simple example, true. But, apply that efficiency to the thousands and thousands of data exchanges every second of every minute of every day, and you can see the huge impact of this.

I know it's crappy timing, from the 30DC perspective, but it will be hugely beneficial to all of us.

Hope that sheds some additional "light" on the Google outage issue for you all.

-Ken

Not exactly sure what you just said, but thank you for saying it!

Cindy

[NextInstinct]

Ken,

would this be having an effect on Gmail? I've had a variety issue
with it for basically, August.

[mitchellg]

It had to be during the TDC.

On the other hand, searches are going fly when this is done.

[kenetrix]

Originally Posted by Caro

WOW Ken!!!

You know your stuff. And I would love to have that 'dumbed down' into simple language as to what it means.

I am gathering you are saying it will be a little quicker to get your search results?

In amongst that - are you also saying that it will enable us to access it for more searches without getting a slap on the wrist for doing too many?

I am impressed - very. Just not 100% clear.

Caro

Glad you found the post useful. I realize it is somewhat "techie", so let me try to "dumb it down" for you all.

Here's the scoop:

Before Protocol Buffers, XML was the primary method use to exchange data amongst servers, applications and systems.

The problem is that XML, being "text based", uses far too much space and bandwidth, and that problem only gets worse, the more extensively it is used.

This problem has always existed with XML, and it always will. XML is a great protocol to use for exchanging data, but it just doesn't scale all that well.

So, imagine an entity like Google. Everything they do, from search to the applications they have rolled out in recent years, totally relies on exchanging data from one entity to another, and that's on a MASSIVE scale, mind you.

So, in short, Google found itself running into the "brick wall", long before anyone else did, and decided that the text-based XML had to be replaced, and took matters into their own hands.

According to the Protocol Buffers documentation, it was initially developed in 2001 to address an index server request/response protocol that Google was having problems with.

Since then, it has become, symbolically, the "lingua franca" for data within Google.

It now has 48,162 message types defined in the Google code tree, across 12,183 .proto files.

That's great, because it shows it's already pretty robustly defined, and it's not just something that they came up with "on the fly" - it's a field-proven, widely used protocol that Google has been using "behind the scenes", in place of XML.

The time has come, however, for Protocol Buffers to expand to the next level, and replace XML within the "Google-sphere", which is what took place this weekend.

If you think about the effort involved in doing that roll-out correctly, it's a scary proposition.

Fortunately, Google has an enormous amount of brilliant software engineers, so they CAN do a roll-out like this successfully, like no one else can.

It is only because of the 30DC that the "public" (i.e. those of us in the 30DC) is even aware of "something going on", due to the tools employed (like Market Samurai, etc.), because we are more "sensitized" to it.

The great news, however, is that the Protocol Buffer roll-out is just about complete, and appears to be a total success.

I am gathering you are saying it will be a little quicker to get your search results?

Well, yes ... but more than a "little quicker": Remember, Protocol Buffers cut transmission time to 1/20th to 1/100th of the time required by XML. So, I'd say it will be a much more than a "little quicker"!

In amongst that - are you also saying that it will enable us to access it for more searches without getting a slap on the wrist for doing too many?

That's hard to say, honestly.

Technically, it has no direct correlation to the Protocol Buffers roll-out, and it's really more of a policy decision from Google's standpoint.

It is, however, perfectly feasible that the Protocol Buffer roll-out COULD have an appreciable influence on any Google relaxation of its policy regarding just what constitutes "too many searches", and thus "warrants a slap on the wrist".

Guess we'll have to just wait and see what the Big "G" decides in this regard.

I hope that made it a bit more understandable, at least as far as the importance of the Protocol Buffer roll-out goes, and its future impact on us all.

If you have any other questions on this, please feel free to ask away, and I'll answer them if I can.

-Ken

[Caro]

Cool bananas - that explanation has been easily absorbed - thank you!

And I am STILL very impressed with your technical knowledge Like...WOW!

Caro

[kenetrix]

Originally Posted by MizzCindy

Not exactly sure what you just said, but thank you for saying it!

Cindy

See my response above. It might make it clearer to you.

-Ken

[kenetrix]

Originally Posted by NextInstinct

Ken,

would this be having an effect on Gmail? I've had a variety issue
with it for basically, August.

I would say "most definitely", since Gmail, as well as the other applications that Google has rolled out, make extensive use of data interchange.

-Ken