Tag - socorro

Entries feed - Comments feed

Tuesday, 2 December 2014

Socorro: the Super Search Fields guide

Socorro has a master list of fields, called the Super Search Fields, that controls several parts of the application: Super Search and its derivatives (Signature report, Your crash reports... ), available columns in report/list/, and exposed fields in the public API. Fields contained in that list are known to the application, and have a set of attributes that define the behavior of the app regarding each of those fields. An explanation of those attributes can be found in our documentation.

In this guide, I will show you how to use the administration tool we built to manage that list.

You need to be a superuser to be able to use this administration tool.

Understanding the effects of this list

It is important to fully understand the effects of adding, removing or editing a field in this Super Search Fields tool.

A field needs to have a unique Name, and a unique combination of Namespace and Name in database. Those are the only mandatory values for a field. Thus, if a field does not define any other attribute and keeps their default values, it won't have any impact in the application -- it will merely be "known", that's all.

Now, here are the important attributes and their effects:

  • Is exposed - if this value is checked, the field will be accessible in Super Search as a filter.
  • Is returned - if this value is checked, the field will be accessible in Super Search as a facet / aggregation. It will also be available as a column in Super Search and report/list/, and it will be returned in the public API.
  • Permissions needed - permissions listed in this attribute will be required for a user to be able to use or see this field.
  • Storage mapping - this value will be used when creating the mapping to use in Elasticsearch. It changes the way the field is stored. You can use this value to define some special rules for a field, for example if it needs a specific analyzer. This is a sensitive attribute, if you don't know what to do with it, leave it empty and Elasticsearch will guess what the best mapping is for that field.

It is, as always, a rule of thumb to apply changes to the dev/staging environments before doing so in production. And to my Mozilla colleagues: this is mandatory! Please always apply any change to stage first, verify it works as you want (using Super Search for example), then apply it to production and verify there.

Getting there

To get to the Super Search Fields admin tool, you first need to be logged in as a superuser. Once that's done, you will see a link to the administration in the bottom-right corner of the page.

Fig 1 - Admin link

Clicking that link will get you to the admin home page, where you will find a link to the Super Search Fields page.

Fig 2 - Admin home page

The Super Search Fields page lists all the currently known fields with their attributes.

Fig 3 - Super Search Fields page

Adding a new field

On the Super Search Fields page, click the Create a new field link in the top-right corner. That leads you to a form.

Fig 4 - New field button

Fill all the inputs with the values you need. Note that Name is a unique identifier to this field, but also the name that will be displayed in Super Search. It doesn't have to be the same as Name in database. The current convention is to use the database name but in lower case and with underscores. So for example if your field is named DOMIPCEnabled in the database, we would make the Name something like dom_ipc_enabled.

Use the documentation about our attributes to understand how to fill that form.

Fig 5 - Example data for the new field form

Clicking the Create button might take some time, especially if you filled the Storage mapping attribute. If you did, in the back-end the application will perform a few things to very that this change does not break Elasticsearch indexing. If you get redirected to the Super Search Fields page, that means the operation was successful. Otherwise, an error will be displayed and you will need to press the Back button of your browser and fix the form data.

Note that cache is refreshed whenever you make a change to the list, so you can verify your changes right away by looking at the list.

Editing a field

Find the field you want to edit in the Super Search Fields list, and click the edit icon to the right of that field's row. That will lead you to a form much like the New field one, but prefilled with the current attributes' values of that field. Make the appropriate changes you need, and press the Update button. What applies to the New field form does apply here as well (mapping checks, cache refreshing, etc. ).

Fig 6 - The edit icon

Deleting a field

Find the field you want to edit in the Super Search Fields list, and click the delete icon to the right of that field's row. You will be prompted to confirm your intention. If you are sure about what you're doing, then confirm and you will be done.

Fig 7 - The delete icon

The missing fields tool

We have a tool that looks at all the fields known by Elasticsearch (meaning that Elasticsearch has received at least one document containing that field) and all the fields known in the Super Search Fields, and shows a diff of those. It is a good way to see if you did not forget some key fields that you could use in the app.

To access that list, click the See the list of missing fields link just above the Super Search Fields list.

Fig 8 - The missing fields link

The list of missing fields provides a direct link to create the field for each row. It will take you to the New field form with some prefilled values.

Fig 9 - Missing fields page

Conclusion

I think I have covered it all. If not, let me know and I'll adjust this guide. Same goes if you think some things are unclear or poorly explained.

If you find bugs in this Super Search Fields tool, please use Bugzilla to report them. And remember, Socorro is free / "libre" software, so you can also go ahead and fix the bugs yourself! :-)

Tuesday, 1 July 2014

About Socorro

I have been working on Socorro for about 3 years now. It was about time that I started talking about it on this blog! With this first post, I am inaugurating 2 new tags, socorro and mozilla. I'm also going to add a language tag to my posts, so each new post will either be en or fr. This way, you can choose more accurately what kind of content you want to read from this blog. The previous links on the tags lead to the filtered Atom feeds, and I will add the mozilla feed to the Planet.

So, what is Socorro?

  1. a planet in the Star Wars universe
  2. a city in Brazil
  3. an island in Mexico
  4. a server to process breakpad crash reports

All of those are true, but as you guessed (I hope! ) we will only consider the last option. So basically, Socorro is a software that collects, processes and stores crash reports generated by breakpad. We use breakpad in our desktop software such as Firefox, Firefox for Android, Thunderbird and FirefoxOS. We then have a Web interface that shows information about the stability our those software, and we call it crash-stats. As usual with Mozilla, most of the information on this site is public, the only exceptions being sensitive data like email addresses, URLs, etc.

Most of Socorro's code base in written in Python. The back-end is raw Python based on a tool called configman, developed by my colleagues Lars and Peter, that handles the configuration and is used as a dependency injection system. It allows Socorro to be very modular, and to add or remove components easily without changing any code. We use 3 different databases:

  1. HBase is the primary storage and is there for data consistency (it contains all the data we receive and process and must never lose data)
  2. PostgreSQL is used as the main source of data for the UI (it contains all the processed data and has a lot of materialized views with various computed information)
  3. elasticsearch is used as the secondary source of data for the UI, mainly for everything related to search. This is what I have been mainly working on.

The Web app is based on Playdoh, Mozilla's customized django framework. We have more and more javascript, mostly using jQuery, but it's a bit chaotic at the moment.

Interested in contributing?

If Socorro sounds like a good project to you and you would like to participate, we would be very happy to hear about you! There are a few places where you can start:

You can also join us in Mozilla's IRC server, channel #breakpad. There are a good variety of bugs to fix on Socorro, from pure front-end work to complex back-end Python problems. And if Socorro is not your thing but you still want to help Mozilla fulfill its mission, you can take a look at whatcanidoformozilla.org!