[Building Sakai] weird performance problem in hibernate
Hedrick Charles
hedrick at rutgers.edu
Wed Feb 19 13:35:59 PST 2014
My biggest problem is that I can’t produce it. I saw it in production, but I have no obvious way of knowing which site it is, though I’m going to try. If I can reproduce it I can try these things. For the moment I’m probably going to bypass Hibernate for one of the other queries of this type. Again, I just need a couple of fields.
I don’t use any collections, so that can’t be an issue. I do use caching. Most of the time is actually spent in the code the checks whether the item is dirty. There are ways to declare a query readonly, which will avoid a lot of that, but for now I think just not using Hibernate makes the most sense. I don’t need any of its features. You’re right that I don’t need either caching or update for the query I’m about to change The one I already changed does updates a few percent of the time. For the moment I’m loading the object only when I need to update it, which is almost never. (In fact I had to gimmick up a test case to try to change code at all.)
I guess my problem is that Hibernate is complex enough that only experts can avoid all of the issues. If I were doing it again I wouldn’t use Hibernate. Not that it’s a bad tool, but that we don’t have anyone with the expertise to use it right.
By the way, I’ve been assuming that each web transaction is done in a different Hibernate session. Is that right?
On Feb 19, 2014, at 3:50:40 PM, Noah Botimer <botimer at umich.edu> wrote:
> This is just a generalization / ballpark idea... When touching lots of "overlapping" items in OSP (for export/stats), we ran into some vaguely similar issues.
>
> In those cases, there were tons of L2 cache checks, etc. Basically, every copy of the same entity in a different list has to be synchronized unless it is evicted from the Hibernate session. This is kind of on the boundary between OLTP and OLAP. More specifically, this kind of operation is read-heavy and is wide and deep, but does not need much in the way of transactional support or isolation. You probably don't care about caching or any updates to the entities, right?
>
> You may be able to improve things with a few techniques. First, I'd look at adjusting the propagation to "supports", which means a transaction will be joined or not required at the service boundary, rather than the usual default "required", which can spawn lots of costly extras. Second, if you really just need the IDs, I'd look at using an HQL query that returns a primitive array, rather than deserializing entities (and registering them with the cache). The third thing (if not doing the second), would be to audit all of the lazy collections in the chain and make sure that anything you traverse is fetched eagerly -- this might dovetail with getting out of the HibernateTemplate model and using the Hibernate Session/Criteria API directly, which is recommended but can be a lot of work.
>
> http://docs.spring.io/spring/docs/3.2.3.RELEASE/spring-framework-reference/html/transaction.html
> http://docs.spring.io/spring/docs/3.2.3.RELEASE/javadoc-api/org/springframework/transaction/annotation/Propagation.html
>
> Thanks,
> -Noah
>
> On Feb 19, 2014, at 2:33 PM, Charles Hedrick wrote:
>
>> Can anyone make suggestions with this?
>>
>> In Lessons, when exporting and copying, I sometimes need to find all items on any page of the site. I use code like this:
>>
>> public List<SimplePageItem> findTextItemsInSite(String siteId) {
>> Object [] fields = new Object[1];
>> fields[0] = siteId;
>> List<String> ids = sqlService.dbRead("select b.id from lesson_builder_pages a,lesson_builder_items b where a.siteId = ? and a.pageId = b.pa\
>> geId and b.type = 5", fields, null);
>>
>> List<SimplePageItem> result = new ArrayList<SimplePageItem>();
>>
>> if (result != null) {
>> for (String id: ids) {
>> SimplePageItem i = findItem(new Long(id));
>> result.add(i);
>> }
>> }
>> return result;
>> }
>>
>>
>> I realie that this isn’t optimal, but it’s also not done very often, so it shouldn’t be an issue. But I recently found a thread that was doing an export, where the equivalent code looking for all text items took hours and didn’t show any sign of ending. I include a stack trace at the end of this message.
>>
>> What I’ve done for that case is avoid hibernate entirely. I just need a couple of fields from the items, so the query returns them and I generate items myself that just have those fields filled in. But I have methods like this returning other sets of items. In principle I can do the same thing for all of them, but do I have to? Anyone have idea idea why this is taking so long? It’s hard to believe that this would ever result in more than a few hundred items. Is there some pitfall I don’t know about in fetching that many items and putting them in a list?
>>
>>
>> java.lang.Class.getInterfaces(Native Method)
>> org.hibernate.intercept.FieldInterceptionHelper.extractFieldInterceptor(FieldInterceptionHelper.java:45)
>> org.hibernate.intercept.FieldInterceptionHelper.clearDirty(FieldInterceptionHelper.java:81)
>> org.hibernate.event.def.DefaultFlushEntityEventListener.isUpdateNecessary(DefaultFlushEntityEventListener.java:209)
>> org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:127)
>> org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:196)
>> org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:76)
>> org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:35)
>> org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:970)
>> org.hibernate.impl.SessionImpl.list(SessionImpl.java:1563)
>> org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283)
>> org.springframework.orm.hibernate3.HibernateTemplate$36.doInHibernate(HibernateTemplate.java:1065)
>> org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
>> org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
>> org.springframework.orm.hibernate3.HibernateTemplate.findByCriteria(HibernateTemplate.java:1055)
>> org.springframework.orm.hibernate3.HibernateTemplate.findByCriteria(HibernateTemplate.java:1048)
>> org.sakaiproject.lessonbuildertool.model.SimplePageToolDaoImpl.findItem(SimplePageToolDaoImpl.java:270)
>> org.sakaiproject.lessonbuildertool.model.SimplePageToolDaoImpl.findTextItemsInSite(SimplePageToolDaoImpl.java:230)
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> java.lang.reflect.Method.invoke(Method.java:606)
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> com.sun.proxy.$Proxy62.findTextItemsInSite(Unknown Source)
>> org.sakaiproject.lessonbuildertool.ccexport.CCExport.outputAllTexts(CCExport.java:607)
>> org.sakaiproject.lessonbuildertool.ccexport.CCExport.download(CCExport.java:1030)
>> org.sakaiproject.lessonbuildertool.ccexport.CCExport.doExport(CCExport.java:214)
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140219/5d1f35b0/attachment.html
More information about the sakai-dev
mailing list