Wicket autocomplete components with remote data

The problem

In web applications you often need to display dropdowns to allow the user to make a selection among a set of choices. There are cases when the set of choices is too large to query and send to the browser all at once.

In this post we will use the example of a database table that stores a list of colleges/institutions across the nation. It contains approximately 30K rows. We want to allow users to select their college using a dropdown. We can see that executing a query to return all 30K colleges, send that data to the browser, and have it render would be too slow.

The solution

One way to solve this problem is using autocomplete or type-ahead fields. They work by requiring the user to enter a minimum number of characters as a search term, and then return some maximum number of results that match that search term.

Let’s look at our simple database table which stores a list of colleges:

CREATE TABLE institution (
  id int(11) NOT NULL AUTO_INCREMENT,
  name varchar(500) NOT NULL,
  status_type tinyint(4) NOT NULL,
  PRIMARY KEY (`id`)
);

You can download an export of the full database with data from GitHub.

Creating the AutocompleteDropDown component

The first thing we’re going to do is create a reusable Wicket component that extends from DropDownChoice. It will have corresponding Javascript code that will automatically be contributed to the page. Let’s jump into the code:

public class AutocompleteDropDown<T extends BaseEntity> extends DropDownChoice<T> {
    @SpringBean private SearchService searchService;
    private static final PackageResourceReference JS_CODE =
            new PackageResourceReference(AutocompleteDropDown.class, "autocomplete_dropdown.js");
    private final SearchType searchType;
    
    public AutocompleteDropDown(String id, IModel<T> model, SearchType searchType, IChoiceRenderer<T> renderer) {
        super(id, model, Collections.emptyList(), renderer);
        this.searchType = searchType;        
    }

    @Override
    public void renderHead(IHeaderResponse response) {
        super.renderHead(response);
        response.render(JavaScriptHeaderItem.forReference(JS_CODE));

        //add the init script
        String suggestionsUrl = (String) urlFor(DropdownSuggestionsPage.class, new PageParameters());
        String initScript = "cd_initAutocompleteDropdown('" + getMarkupId() + "', '"+suggestionsUrl+"');";
        response.render(OnLoadHeaderItem.forScript(initScript));
    }

    @Override
    protected void onComponentTag(final ComponentTag tag) {
        super.onComponentTag(tag);
        tag.put("search-type", searchType.toString());
        tag.put("data-minimum-input-length", String.valueOf(getMinCharacters()));
        tag.put("search-filters", EntityUtil.objectToJson(getFilters()));
    }

    protected int getMinCharacters() { return 3; }
    protected AutocompleteFilters getFilters() { return null; }

    @Override
    public IModel<? extends List<? extends T>> getChoicesModel() {
        T obj = getModel().getObject();
        return new ListModel<>(obj == null ? Collections.emptyList() : Collections.singletonList(obj));
    }

    @Override
    protected T convertChoiceIdToChoice(String id) {
        if(StringUtils.isBlank(id)) {
            return null;
        }

        Integer pkId = null;
        try {
            pkId = Integer.parseInt(id);
        } catch (NumberFormatException e) {
            return null;
        }

        Optional<T> found = searchService.getById(searchType, pkId);
        return found.isPresent() ? found.get() : null;
    }
}

There are a few things we should note here:

• The constructor accepts a “search type” that represents all the different types of business objects that can be searched for in our application: colleges, users, photos, etc.
• The renderHead method is overridden to contribute our Javascript code to the page as well as an initialization script that uses the component’s markup ID.  The  suggestionsUrl is the page that will accept the search term and return suggestions.  We will cover it in more detail below.
• The onComponentTag method is overridden to render HTML attributes into our <select> tag which are than passed to the suggestions URL.
• The convertChoiceIdToChoice method is overridden to convert an integer representing the ID of the selected college into an Institution object.  A regular DropDownChoice does this by first getting all the choices, and using the position in the list of choices, returns the selected choice. 
  • This would not be suitable for our component because it needs to rely on a database ID to identify the user’s selection and cannot retrieve a list of all the choices.
• The getChoicesModel method is overridden so that when rendering the page, and a selection has already been made, the dropdown will be rendered with the user’s selection pre-selected.

---

Adding the Javascript code to make it all work

Next we will add the Javascript code referenced in JS_CODE PackageResourceReference. When our component is added to the page, it will use the function defined in this script, cd_initAutocompleteDropdown, to initialize itself:

function cd_initAutocompleteDropdown(inputId, suggestionsUrl) {
    $("#" + inputId).select2({
        placeholder: "Start typing...", allowClear: true, width: "100%",
        ajax: {
            url: suggestionsUrl, dataType: "json", type: "post", delay: 250,
            data: function (params) {
                params.page = params.page || 1;
                return {
                    term: params.term,
                    page: params.page,
                    searchType: $(this).attr("search-type"),
                    filters: $(this).attr("search-filters")
                };
            },
            processResults: function (data, params) {
                return {
                    results: data.results,
                    pagination: {
                        more: (params.page * 15) < data.count
                    }
                };
            }
        }
    });
};

We use the awesome Select2 library here. As the user types, ajax requests are sent to the suggestions URL with the search term and the suggestions are sent back.

The full code for our component can be found here.

Creating the “suggestions” endpoint

The next thing we need to do is create a Wicket WebPage that will receive a search term, search type, and other information, and return suggestions in JSON format. Let’s take a look at it:

public class DropdownSuggestionsPage extends JsonRequestPage {
    private static final long serialVersionUID = 1L;

    private static final int COUNT_PER_REQUEST = 15;
    @SpringBean private SearchService searchService;

    public DropdownSuggestionsPage(final PageParameters pp) {
        super(pp);
    }

    @Override
    protected String sendResponse(PageParameters pp) {
        IRequestParameters reqParams = getRequest().getRequestParameters();
        SearchType st = getSearchType(reqParams);
        String term = getSearchTerm(reqParams);
        int page = getPageNum(reqParams);
        AutocompleteFilters filters = getFilters(reqParams);

        if (st == null || !isTermOk(term)) {
            return getEmptyResponse();
        }

        switch (st) {
            case INSTITUTIONS:
                return getInstitutions(term, page, filters);
            case USERS:
                return getUsers(term, page, filters);
            default:
                return getEmptyResponse();
        }
    }

    private boolean isTermOk(String term) {
        return term != null && term.length() > 0;
    }

    private String getInstitutions(String term, int page, AutocompleteFilters filters) {
        List<Institution> fullList = searchService.searchInstitutions(term, filters);
        if (fullList.isEmpty()) {
            return getEmptyResponse();
        }
        return getStringResults(fullList, UIHelpers.getInstitutionChoiceRenderer(), page, COUNT_PER_REQUEST);
    }

    private String getUsers(String term, int page, AutocompleteFilters filters) {
        List<ComplexUser> fullList = searchService.searchUsers(term, filters);
        if (fullList.isEmpty()) {
            return getEmptyResponse();
        }
        return getStringResults(fullList, UIHelpers.getComplexUserChoiceRenderer(), page, COUNT_PER_REQUEST);
    }

}

Some things to note about this code:

• We first get the search type, search term, the page number of results, and other filters from the request POST body.  We validate that the search type and search term is present.
• We then check the search type to determine what type of suggestions we should return.
• We implement methods that handle each search type that will query the database.
• We return the results in JSON format which will be used in cd_initAutocompleteDropdown.processResult that we saw above.

---

The fun part!

Now let’s see how easy it is to use this component in our Wicket pages. On the HTML side all we need to do is include the Select2 library, and our <select> element does not change. In our WebPage we create the component like this:

AutocompleteDropDown<Institution> field = new AutocompleteDropDown<>("selectInstitution", objModel(this::getSelectedInstitution, this::setSelectedInstitution), SearchType.INSTITUTIONS, UIHelpers.getInstitutionChoiceRenderer());

We pass the SearchType so that our component knows what type of suggestions to give and a IChoiceRenderer that knows how to display those suggestions. Otherwise we can use this just like any other Wicket DropDownChoice. When you add this to your page and view the page source, you will see a small script added to the <head> of the page that will look something like:

<script type="text/javascript" >
/*<![CDATA[*/
Wicket.Event.add(window, "load", function(event) { 
cd_initAutocompleteDropdown('selectInstitution1', './org.coderdreams.webapp.autocomplete.DropdownSuggestionsPage');;
;});
/*]]>*/
</script>

This comes from the renderHead method of our component that allows it to automatically initialize itself.

A bonus

You may have noticed a reference to AutocompleteFilters in our component which we glossed over. The idea is that you may want to filter suggestions by more than just a search term. For example you may want to have one dropdown for searching all colleges, and a second dropdown for searching only colleges with a status of pending.

That is the purpose of the AutocompleteFilters class, it contains all the different options for filtering suggestions. Once you create an instance of the class, it gets serialized into JSON and stored as an HTML attribute of our component. When the user types in a search term, the filters get sent to our suggestions page and are used there in addition to the search term.

The way you would do this is quite simple. Let’s modify the above example to create a component that searches only colleges that are pending:

AutocompleteDropDown<Institution> searchPendingField = new AutocompleteDropDown<>("selectPendingInstitution",
	objModel(this::getSelectedPendingInstitution, this::setSelectedPendingInstitution), SearchType.INSTITUTIONS, UIHelpers.getInstitutionChoiceRenderer()) {
    @Override
    protected AutocompleteFilters getFilters() {
	return new AutocompleteFilters().setStatusType(StatusType.PENDING);
    }
};

Now if we look at the page source again, we will see how the filters are rendered into the HTML tag:

<select class="form-control" wicket:id="selectPendingInstitution" name="selectPendingInstitution" id="selectPendingInstitution2" search-type="INSTITUTIONS" data-minimum-input-length="3" search-filters="{"skip":0,"maxResults":30,"statusType":"PENDING"}">
</select>

You can check out the full test page with usages examples here.

---

More ideas

You can use reuse this component for all of your business objects with minimal changes. All you would need to do is define a new search type and a new method into the suggestions page to handle returning the appropriate results.

You can use this anywhere you need to selections with choices coming from the database where it would be infeasible to display all the choices. Some ideas are user profiles, blog entries, photos, friend selectors, etc.

In the next post we will look at how to improve the suggestions using fuzzy search.