voyent
Lazy loading / Data on demand / Paging big tables solution  XML
Forum Index -> General Help Go to Page: 1, 2 Next 
Author Message
jonsi

Joined: 15/May/2008 00:00:00
Messages: 2
Offline


Hi all, I hope this post will ease people the pain when it comes to working with huge data sets. I've done allot of searching, googling and here I present my findings that will hopefully save you some hours. Credits go to Roger at www.ilikespam.com. I highly recommend that this sort of stuff should be well documented in IceFaces tutorials and developers guides (untill the planed <ice:richTable> component will come to existence in IceFaces 2.0)

The problem: IceFaces objective is to provide enterprise developers some handy tools when creating some web apps (and does a damn good job at it imo). I (and most likely many others) however found one stone left unturned, in enterprise development it is very common that you have to work with queries that return huge data sets. When you use a ice:dataTable and a ice:dataPaginator to presents this data you'll have to provide the ice:dataTable with a list or an array of objects (business logic objects most of the time). As in my case, creating and returning a list with few thousand entries was simply not an option and probably not for anyone (unless you've got 1337 TB worth of RAM).

Solution: I came across 2 solutions, one much better than the other. I'll not go into much detail of the first one but here's the link to that forum post (note: I did try this one out but couldn't get it to work for me). The other solution is much more simple and makes perfect sense once you realize it. In essence you simply create a new class (LazyLoading.java for example) and make it extend java.util.AbstractList. What you do then is simply "lie" how big the list is because the ice:dataPaginator uses the size() method in the list to find out the entries count. Then you implement your own get(int index) method which loads data as they become needed (when you flip the page). A link to the page with the second method (the elegant and simple one) can be found here!

Exmple of how it works: You've got a page with a <ice:dataTable> and a <ice:dataPaginator> assigned to it with 10 items per page. Your backingbean has a instance of LazyLoadingList (lazyList) which you initialize with a query object, page size and the number of results the query should have returned (this means in fact that you have to do 2x queries every time, however the count(*) query is usually very lite). In the backingbean a instance of LazyLoadingList is returned to the dataTable. The page loads and the LazyLoadingList makes a query for the first 10 items (IceFaces calls lazyList.get(0), ..., lazyList.get(9) to fill in the table). Then lets say you press any page (other than the first) in the dataPaginator, that triggers IceFaces to refresh the dataTable and to fetch new items (somewhat in the lines of lazyList.get(10) for the first item on the second page). The LazyLoadingList get(int index) method checks if it has the requested indexes in range, if it doesn't (a new page) it does a query for it.

My way: Now I did it somewhat different than on the web page I referred to but it was a great starting point for me. What I did different was I made a interface LazyLoadable.java that has the method getUpdatedList(int index, int pageSize) that returns a list and I make my backingbean implement that method. This is the method that actually queries the databae. I do this because I've already got my database related objects initialized there instead of creating them again or passing them as parameters (whatever suits you). In the constructor of LazyLoadingList I take LazyLoadingList(LazyLoadable lazyObject, int pageSize, int numResults). Then instead of a HashTable I used 2x lists. When using this I've a common refresh method in my backingbean (column sort, filter change etc.). So every time something changes this method gets called and in it I recreate the list as follows:

  • this.businessObjectList = new LazyLoadingList(this, dataTable.getRows(), myQueryObject.count());

    dataTable: is a global HtmlDataTable instance so I define how many rows I want in the .jspx document. myQueryObject.count(): returns how many results the query really returns (could be couple of thousand or whatnot). In addition to this I've the usual getBusinessObjectList() method that gets wired something like this (note: just an example, in practise I've a lot more of them): < ice:dataTable rows="10" value="#{backingbean.businessObjectList}" >

    I'll attach those 2x files as attachment and I belive the code speaks for itself (the comments were in Icelandic so I removed them but I've described everything in English here in this post). Feel free ask around and we'll see if I can be of any help. Cheers from Iceland (kinda appropriate we use IceFaces and we're from Iceland don't you think?) -_-


    Keywords: lazy loading data on demand paging big tables huge query datapaginator paging
     Filename LazyLoadable.java [Disk] Download
     Description The complementing interface
     Filesize 118 bytes
     Downloaded:  1772 time(s)

     Filename LazyLoadingList.java [Disk] Download
     Description Custom List.
     Filesize 1 Kbytes
     Downloaded:  1795 time(s)

  • nerlijma

    Joined: 22/Oct/2007 00:00:00
    Messages: 36
    Offline


    Hi, I have tried you example (very nice) and pagination works fine. However, with 10 million of rows and a RowSelector, when selecting an item it takes for ever.

    If you have a look at RowSelector.java (309) where

    Code:
     // ICE-2024: should clear the whole table, not just the displayed page.
     if (!getMultiple().booleanValue()) {
        for (int i = 0; i < dataTable.getRowCount(); i++) {
       if (i != rowIndex) {
       dataTable.setRowIndex(i);
       setValue(Boolean.FALSE);
       }
       }
      }


    you can see that ICEFaces is iterating all 10 millon of rows to clean the selected value because is calling getRowCount().

    Are you selecting a row in your table and it is selected properly?
    if not, have you any workaround for this?


    jonsi

    Joined: 15/May/2008 00:00:00
    Messages: 2
    Offline


    Sup nerlijma. We checked it out and you're right.

    We are now working on new project where we use a the ice:rowSelector component and uppon inspection we discovered that the RowSlector indeed iterates over the entire list (the LazyLoadingList fetches new rows as they become needed). Not only does it iterate the entire list, it does it quite often (we haven't found a pattern yet but it iterates the list 2 or 3 times). Along with this we did some debugging and also found out that when IceFaces fills in the table it fetches every page 2 times.

    Atm we're trying to figure it out and will post our findings here. Those two projects we're working on were sortof a tryout for IceFaces, while the concept is nice its those practical kinks that makes us less and less likely to adopt it.

    cheers, jonsi
    rdefuentes

    Joined: 11/Jul/2008 00:00:00
    Messages: 1
    Offline


    HI all,

    Rowselector is not working for me with this lazy pattern too, as nerlijma posted, it is iterating the whole list.

    I will post a very simple example to the ICEfaces team to have a look. Huge tables is a quite often requirement !!

    best regards
    mark.collette


    Joined: 07/Feb/2005 00:00:00
    Messages: 1692
    Offline


    One work-around is to declare your rowSelector as using multiple selection, and then in your selectionListener, enforce the single selection yourself. This is easiest done when using a smart Map to keep your selections, instead of using a per row field for selection.
    [Email]
    nerlijma

    Joined: 22/Oct/2007 00:00:00
    Messages: 36
    Offline


    Mark,

    I have changed RowSelector.java adding a lastRowIndex used only if is not multiple. Also commented where it iterates over all rows and uncommented the old code, where if is not the selected row, rowselector marks it as FALSE.
    Therefore, if is not multiple, with every click, old selected index is set to FALSE as well as not clicked ones. Only clicked row index is set to true.
    I am using 16 Million rows dataTable, with a LazyList pattern, and works perfect.

    Code:
             if (!getMultiple().booleanValue()) {
                     dataTable.setRowIndex(lastClickedRow);
                     setValue(Boolean.FALSE);
                     lastClickedRow = rowIndex;
             }
                     
                     // ICE-2024: should clear the whole table, not just the
                     // displayed page.
     //                if (!getMultiple().booleanValue()) {
     // this code all commented
     //                }
     
                 } else {
                     // Uncommented this...
                     if (Boolean.FALSE.equals(rowSelector.getMultiple())) {
                         // Clear all other selections
                         rowSelector.setValue(Boolean.FALSE);
                     }
                 }
    bchi49

    Joined: 06/Sep/2008 00:00:00
    Messages: 1
    Offline


    Here's a generic version, please review:



    public class LazyLoadingList<E> extends AbstractList<E> {
    private LazyLoadable<E> lazyObject;
    private List<E> firstList;
    private List<E> viewList;
    private int currentPage = -1;
    private int numResults;
    private int pageSize;

    public LazyLoadingList(LazyLoadable<E> lazyObject, int pageSize, int numResults) {
    this.lazyObject = lazyObject;
    this.numResults = numResults;
    this.pageSize = pageSize;
    }

    @Override
    public E get(int i) {
    if (i < pageSize) {
    if (firstList == null)
    firstList = lazyObject.getUpdatedList(i, pageSize);
    return (E)firstList.get(i);
    }
    int page = i / pageSize;
    if (page != currentPage) {
    currentPage = page;
    viewList = lazyObject.getUpdatedList(i, pageSize);
    }

    return viewList.get(i % pageSize);
    }

    @Override
    public int size() {
    return numResults;
    }

    public void setNumResults(int numResults) {
    this.numResults = numResults;
    }

    }







    public interface LazyLoadable<E>
    {
    public List<E> getUpdatedList(int index, int pageSize);
    }
    kumareshav

    Joined: 10/Apr/2008 00:00:00
    Messages: 44
    Offline


    can i have some examples with this lazyloading class being used with data table?

    Regards,
    Kumaresh
    [Email]
    trumaxformax1

    Joined: 01/Mar/2008 00:00:00
    Messages: 3
    Offline


    Hi.

    I need to implement some method to show big tables too. Reading the comments I not sure If you try to implement a paging or something like a live-scroll.

    I have found a component in http://www.nitobi.com/products/completeui/demos/explorer/ which show what I need. No matter how many records you have it show as you scroll down/up....

    My question is: Work you example as this???

    Thanks.

    kumareshav

    Joined: 10/Apr/2008 00:00:00
    Messages: 44
    Offline


    I will get back to u once i finished look into this site which you have given.

    Thanks

    Regards,
    Kumaresh
    [Email]
    trumaxformax1

    Joined: 01/Mar/2008 00:00:00
    Messages: 3
    Offline


    Ok, thanks.


    I'll be waiting.


    Greetings
    brad.kroeger

    Joined: 26/Oct/2004 00:00:00
    Messages: 298
    Offline


    I have completed a tutorial covering lazy loading a dataTable with JPA and the ICEfaces rendering api. It is located at http://anonsvn.icefaces.org/repo/tutorials/trunk/tutorials/samples/dataTable-JPA.

    Brad Kroeger
    Developer
    ICEsoft Technologies, Inc.
    [Email]
    brad.kroeger

    Joined: 26/Oct/2004 00:00:00
    Messages: 298
    Offline


    Here is the proper link, the docs folder has the html page with the tutorial content. The tutorial should be published to icefaces.org in the near future. http://anonsvn.icefaces.org/repo/tutorials/trunk/tutorials/samples/dataTable-JPA

    Brad Kroeger
    Developer
    ICEsoft Technologies, Inc.
    [Email]
    ybendek

    Joined: 21/Dec/2008 00:00:00
    Messages: 3
    Offline


    Hi all.

    well I'm trying Jonsi option to paging a lot of records, it works fine...however when I tryied to log what is the real query running with the database... something happen :S.

    I have an DAO object implementing LazyLoadable interface, checking what is the real request I have some log in it to catch every SQL request.. I don't know why.. but every page refresh is requesting all data many times ..and all per groups... let me explain me with my log...

    09/01/07 15:32:10 index [0] pageSize [10]
    09/01/07 15:32:10 index [10] pageSize [10]
    09/01/07 15:32:10 index [20] pageSize [10]
    09/01/07 15:32:10 index [30] pageSize [10]
    09/01/07 15:32:10 index [40] pageSize [10]
    09/01/07 15:32:10 index [50] pageSize [10]
    09/01/07 15:32:10 index [60] pageSize [10]
    09/01/07 15:32:10 index [70] pageSize [10]
    09/01/07 15:32:10 index [80] pageSize [10]
    09/01/07 15:32:10 index [90] pageSize [10]
    09/01/07 15:32:10 index [100] pageSize [10]
    09/01/07 15:32:10 index [10] pageSize [10]
    09/01/07 15:32:10 index [20] pageSize [10]
    09/01/07 15:32:10 index [30] pageSize [10]
    09/01/07 15:32:10 index [40] pageSize [10]
    09/01/07 15:32:10 index [50] pageSize [10]
    09/01/07 15:32:10 index [60] pageSize [10]
    09/01/07 15:32:10 index [70] pageSize [10]
    09/01/07 15:32:10 index [80] pageSize [10]
    09/01/07 15:32:10 index [90] pageSize [10]
    09/01/07 15:32:10 index [100] pageSize [10]
    09/01/07 15:32:10 index [10] pageSize [10]
    09/01/07 15:32:10 index [20] pageSize [10]
    09/01/07 15:32:10 index [30] pageSize [10]
    09/01/07 15:32:10 index [40] pageSize [10]
    09/01/07 15:32:10 index [50] pageSize [10]
    09/01/07 15:32:10 index [60] pageSize [10]
    09/01/07 15:32:10 index [70] pageSize [10]
    09/01/07 15:32:10 index [80] pageSize [10]
    09/01/07 15:32:10 index [90] pageSize [10]
    09/01/07 15:32:10 index [100] pageSize [10]
    09/01/07 15:32:10 index [10] pageSize [10]
    09/01/07 15:32:10 index [20] pageSize [10]
    09/01/07 15:32:10 index [30] pageSize [10]
    09/01/07 15:32:10 index [40] pageSize [10]
    09/01/07 15:32:10 index [50] pageSize [10]
    09/01/07 15:32:10 index [60] pageSize [10]
    09/01/07 15:32:10 index [70] pageSize [10]
    09/01/07 15:32:10 index [80] pageSize [10]
    09/01/07 15:32:10 index [90] pageSize [10]
    09/01/07 15:32:10 index [100] pageSize [10]

    I have just 107 records and 10 records per page, I'm not using rowselector.. is just a plane table with pagination... please help :'(
    meeanaalenoor

    Joined: 15/Jan/2009 00:00:00
    Messages: 11
    Offline


    How the LazyLoadableList Managed bean is processed in the .jspx page ?? Can you give me the code.
     
    Forum Index -> General Help Go to Page: 1, 2 Next 
    Go to:   
    Powered by JForum 2.1.7ice © JForum Team