Conceptual question about paging

I really liked the paging lesson… has a lot of real world application.

I learned something through the lesson… it is the concept of what is a request compared to what is on the view in the UI…

Specifically, even though a query is made for additional movies the request isn’t cumulative it is soley based on the request batch set i.e. 20 at a time. How is this known by the view? I can’t find the source code directly responsible but i am assuming there is an array and it is pushing via the controller.

Question is, what is being specifically done on the frontend that maintains that view state. i.e. all of the requested data remains whiles new data requests are added. Again, i imagine pushing the new data to an array but i’d like to confirm.

The front end is working with ReactJS, that’s who create all the magic. Data binding

I really like to work with VueJS.

What you will see if you look in the /src/api/movies.controller.js is that both the functions for apiSearchMovies() and apiFacetedSearch() have this same line of code:

page = req.query.page ? parseInt(req.query.page, 10) : 0

What that is doing is inspecting the “query parameters” as passed to the URI with something like:

/api/v1/movies/search?text=Heist&page=7

So both text and page are valid properties of the req.query, which would be basic information from the front end being partly from ‘state’ being the search term you entered in before, but of course would be returned as explained in a moment, and of course the “page number” which would typically come from clicking on a “paging” link that actually triggered the request.

You can note that again both of those functions actually return a similar response object:

let response = {
  movies: moviesList,
  page: page,
  filters,
  entries_per_page: MOVIES_PER_PAGE,
  total_results: totalNumMovies,
}

res.json(response)

The facets version of course returning additional information for the “facets” of course.

Basically that breaks down to:

  • movies: An array of returned results, which will be up to a maxium lenght as defined by another item in the response.
  • page: Which indicates the current page of results retrieved. This should marry up with the page number sent in the request.
  • filters: Shows the actual “query” filter applied. Depending on whether it was “free text search”* or facets there will be different information in here. Basically this will line up with the input of the other “query parameters” on the request.
  • entries_per_page: Is actually HARDCODED in the implementation shown here. More typically you would have a similar “query parameter” to allow the UI to select this.
  • total_results: Note that particularly since this is paging, there will be in the actual DAO implementation a second query being issued with the same conditions, but none of the skipping or limiting of results, so you know how many matches there were for a given set of conditions.

These are all basically passed through from their respective DAO methods, where the real magic actually happens, and the part you were/are required to complete in the exercises here:

static async getMovies({
    // here's where the default parameters are set for the getMovies method
    filters = null,
    page = 0,
    moviesPerPage = 20,
} = {})

static async facetedSearch({
    filters = null,
    page = 0,
    moviesPerPage = 20,
} = {})

Noting that the signatures are basically the same, so it’s the controller* part which is essentially following different rules in order to construct the filters argument for each of those. And as noted the page is simply a pass-through of whatever was requested.

If you want to know how these map together, then that is what the /src/api/movies.route.js is doing, and also similarly for the other “routes” of the back-end API.

1 Like

Actually all the "magic" is really happening in the back-end, and that’s basically the point of the course.

The “front-end” design is pretty much standard Redux pattern fare, issuing actions then retrieving data and changing state so that “components” ( the only React part really ) can display data from state.

Not really a lot happening, but it does serve as a lesson that your logic really does not belong in the front-end, which should basically only be concerned with “presentation” of data through interaction. “Heavy Lifting” always belongs to the place where the common logic can be shared.

I will concede though that the onscroll() paging might seem a “little magical”. All the same as above where the main work is being done, just appending the array of movies onto the existing list in state:

Scrolling implementation

  onScroll() {
    const scroll = document.getElementById("root")
    if (
      !this.props.movies.paging &&
      document.body.offsetHeight + window.pageYOffset >=
        scroll.scrollHeight - 1500 &&
      this.props.movies.movies.length !== this.props.movies.total_results
    ) {
      this.props.movieActions.beginPaging()
      this.props.movieActions.paginate(
        this.props.movies.movies,
        this.props.movies.page,
        this.props.movies.filters
      )
    }
  }

And paginate actually just calls the backend as described:

export funtion paginate(currState, currPage, filters) {
  return dispatch => {
    let query
    let url
    if (Object.keys(filters).length !== 0) {
      query = Object.keys(filters).reduce(
        (acc, curr) => [...acc, `${curr}=${filters[curr]}`],
        [],
      )
      query = "?" + query.join("&") + `&page=${currPage + 1}`
    } else {
      query = `?page=${currPage + 1}`
    }
    if (Object.keys(filters).includes("cast") && useFacets) {
      url = `/api/v1/movies/facet-search${encodeURI(query)}`
    } else {
      url = `/api/v1/movies/search${encodeURI(query)}`
    }
    return request(url, {
      method: "GET",
      mode: "cors",
    })
      .then(json =>
        dispatch(receivedPagination(currState, currPage, json, dispatch)),
      )
      .catch(e => dispatch(fetchMoviesError(e.message)))
  }
}
2 Likes

@neillunn has provided an excellent answer.

The UI is parroting values sent to it from the API back to the API, incrementing the page value by 1. It has no notion of the results cursor and no notion of where a movie might be within a set of documents. It (almost blindly) trusts the API to have returned a new set of documents it should add to the documents it has in state, and then displays those. We happened to choose React, but this could have been done in Vue or Angular.

This allows us to focus on more fun things in the UI like what happens when you click on one of the Matrix movies to look at the movie detail…

Thanks for taking the time to answer. This is has excellent real world application.

Keep in mind we’re using a “simpler” form of paging that isn’t as performant as an alternative! We opted not to expose the other method in this course due to having to have something to order by. As a general rule, do not rely on natural ordering but on some other field with a determined order instead.

Forgot the earlier conversation on the actual “back end” paging implementation earlier. As @nathan.leniz brings up, it would indeed be more efficient for this application to use a 'ranged query" instead of skipping since the former can use an index where skip() does not.

Though slightly shameless self promotion :grinning:, there is I believe a more detailed write up than that link @nathan.leniz gave ( which is why I posted a Google Search link earlier ) considering there are “actual coded examples” rather than pseudocode.

But now the for shameless plug:

The reason this applies to the “mflix” design in particular is the only place any “paging” is implemented in the onScroll(). This is often referred to as infinite scroll based on the effect of scrolling the the seemingly infinite crap information often found in social feeds. You are basically always “moving forwards” hence the reference to “forward paging” in that write up.

By contrast, more traditional approaches with “page controls” like Google Search, or Stack Overflow actually need that skip() approach in order to allow you to “jump” from page 2 to page 10 at any time. It’s a slower approach as described, no matter which database system is used. Though such big sites often employ other optimizations by pre-calculating page groupings for common ordering, but that’s really an entirely different and very broad topic to explain.