This has certainly been done before....but here is a way to process objects from a Java Collection (ArrayList, Vector, etc.) and to work with a group of objects in arbitrary batch sizes.  For example, this code snippet below may be useful if you need to run a massive data manipulation operation and are concerned about server time-outs, connections being lost, etc.  So one would run smaller batches to prevent data corruption and problems related to system limitations. When you want to execute data manipulations in batch mode it's not all that complicated but the snippet below may help in desiging Java code. 

This particular included sample is to assign a lifecycle (dm_process) to a select group of documents that come into the Method via an array list. So, data (object id's) are contained in the array list.  A Documentum Method is created and one passes the batch size parameter to the Java Method.  This permits easy modification to batch number while running the data manipulation Java classes.  Obviously the snippet below is part of a larger set of Java Classes & Methods for setting lifecycle.  But, as I state above, this is a generic batch processor that can be  used in any kind of data processing using Java.

1. boolean applyIt(ArrayList docIds,int batchSize,Map params){

2. int wholeNumBatches = getWholeNumBatch(docIds.size(),batchSize);

3. int modU = getModulus(docIds.size(),batchSize);

4. DfLogger.debug( "LM","DoApplyDLC.applyIt...WILL PROCESS " + wholeNumBatches + " WHOLE BATCHES AND A PARTIAL BATCH OF " + modU + " DOCUMENTS.",null, null);

5. int m = 0;

6. int t= batchSize;

7. for(int p=0;p<wholeNumBatches;p++){

8. for(int k=m;k<t;k++){

9. Object o = docIds.get(k);

10. String id = o.toString();

11. IDfSysObject sysObj = getSys(id,params);

12. try {

13. String typeName = sysObj.getTypeName();

14. IDfId policyId = fetchPolicyId(typeName,params);

15. sysObj.attachPolicy(policyId,"WIP",null);

16. DfLogger.debug("LM","DoApplyDLC.applyIt...set lifecycle for document : " + sysObj.getObjectName(),null, null);

17. } catch (DfException e) {

18. DfLogger.debug("LM","DoApplyDLC.applyIt...exception getting type name : " + e,null, null);

19. e.printStackTrace();

20. }

21. }

22. }

24. m = m + batchSize;

25. t = t + batchSize;

26. }

27. //process partial batch, if there are any docs in a partial batch

28. if(modU>0){    ...........

29. } 

31. }

Interpretation:

Line 1 shows that you want to pass in a Collection object, and need batch size to come from a parameter or other source. 

Line 2 gets the number of whole batches to be processed. They way this Method works is that 1st whole batches are processed, then a single partial batch is processed.

Line 3 involves fetches the Modulus, which can be done something like this in a separate method. This is used for partial batch. This part is not shown above but would commence at Line 28.

Line 5 making new variable to use in initiating & counting - more about this later

Line 6 setting a new variable t to Batch Size - more about this later

Line 7 start of outside loop.  This outside loop runs for p, which is the number of whole batches you have. For example, if you have 1000 objects in a Collection and a batch size of 100, p will be 10.

Line 8 start of inner loop....variable m starts at 0, but is always increased by the batch size, k runs for number of times of batch size

Line 9 get an object of data at correct location from Collection

Lines 10-23 This is application specific and in this case involves changing an object to a string, getting what's called a sysobject from system, applying lifecyle. But any particular operation could be done here on the data fetched from Collection.

Line 24 set m to m + batch size to keep counter indexing correct location in Collection. Note this is outside inner loop so will keep reading down in the Collection past where the 1st batch already accessed.

Line 25 set t to t + batch size. Here again, this keeps indexing number "k" reading at correct location in Collection; otherwise with more than one whole batch Collection would be read again from 0 to the end of the batch.

Line 28. As stated above this modulus indicated number of objects in Collection "left over" to be processed as a single partial batch.  One can now do the same processing  to the partial batch if that is the requirement. 

Sometimes it is really useful to immediately test a document's ability to be fulltext-indexed on a FAST server.  It is possible, of course, to submit the document again and again through the Documentum Administrator (DA) interface. If document has failed you can right click the document (from DA) and click Resubmit. But then you have to wait until the Index Agent listener picks up the document for indexing and this is more time consuming. You can also cause a document resubmission by editing or checking and checking out but the solution using api below is fastest.

To submit a document right away: first get its object id and then go into API Testor in DA or IAPI32.exe to submit this short statement:

queue,c,090008468000289a,dm_fulltext_index_user,dm_force_ftindex,,,,debug

A second advantage to submitting above api command with "debug" option is that you get valuable information about the document AFTER it's fulltext indexed.  If you submitted the above command go to this location to locate debugging output:

...\Documentum\bea9.2\domains\DctmDomain\servers\DctmServer_IndexAgent1_DCTMFAST\data\IndexAgent1\

export\IndexAgent\dctmFAST_IndexAgent1\content\debug\090008468000289a_1

Sorry about the path...of course they have to make you look for the output!   Remember, some of your server names and index agents will be spelled differently than above; those parts in italics are variable.

There is one interesting file in this location now: an XML file with a listing of most (or all) of the attributes that will be indexed. In other words, the XML file contains all the information on attributes it is sending to the index.  See Fig. 1 below.

How are the attributes arranged inside the XML file?  Below is a small part of the document...enough to hopefully get you interested. As you can see FAST Instream is converted attribute values into XML elements and classified them by types (date, string, etc.). 

In summary, this technique for debugging a single document by forcing fulltext indexing through API is a useful way to debug entries if you have customized fulltext indexing or if you are troubleshooting a problem relating to metadata indexing.

<?xml version="1.0" encoding="UTF-8"?>

<dmftdoc dmftkey="090008468000289a">

    <dmftkey>090008468000289a</dmftkey>

    <dmftmetadata>

        <dm_document>

            <r_object_id dmfttype="dmid">090008468000289a</r_object_id>

            <object_name dmfttype="dmstring">Product Support Matrix.pdf</object_name>

            <r_object_type dmfttype="dmstring">dm_document</r_object_type>

            <r_creation_date dmfttype="dmdate">2008-10-23T14:17:54</r_creation_date>

            <r_modify_date dmfttype="dmdate">2008-10-23T14:17:54</r_modify_date>

            <r_modifier dmfttype="dmstring">Steve Garrison</r_modifier>

            <r_access_date dmfttype="dmdate">2008-10-23T08:19:16</r_access_date>

 XML file continues

Figure 1. Image of directory where debugging files stored

If you want a Java application to run daily, hourly or on some other frequency and the server is Windows, this entry provides a step-by-step method for configuring such a system.  Below are the basic steps in brief:

1. write java class or obtain classes to use
2. move class or classes to root location:  java_home\bin (where java.exe is found)
3. put your classes in the file path descending from bin, as usual (i.e. com.utilities.mycompany)
4. put .properties file (if any) in bin directory (java_home\bin )

5. create a batch file on desktop or anywhere.  Inside batch file place your java class invocation
6. point to batch file in Windows Scheduler.

7. Configure Windows Scheduler to run on some schedule.
 
Details of Process (uses same sequence as above):

1. write java class or obtain classes to use. Can also use jar files, of course.

2. move class or classes to root location:  java_home\bin (where java.exe is found). If you don't have java_home environment variable set search for java.exe and use the JRE/JDK folder that matches the version of Java JVM you're using. It's often found in c:\program files\java...

3. put your classes in the file path descending from bin, as usual (i.e. com.utilities.mycompany).  See image below: 

4. put .properties file (if any) in bin directory (java_home\bin ). In this example: C:\Program Files\Java\jdk1.5.0_09\bin

5. create a batch file on desktop or anywhere.  Inside batch file place your java class invocation.  Here is a sample batch file. Of course, you can get as fancy and include as many features as you want. Check the Internet for details about how to write batch (.bat) files...this is an entire subject in itself:

echo Running Java Move File utility
echo %date% %time% Start of Running Java Move File utility> c:\test.log
cd C:\Program Files\Java\jdk1.5.0_09\bin
java -cp ".;" com.util.cmmc.CopyAndDelete 1>>c:\test.log 2>>&1
echo Batch file completed

Intrepretation of bat file (in parentheses)

echo Running Java Move File utility (just a comment)

echo %date% %time% Start of Running Java Move File utility> c:\test.log  (this line puts a timestamp and title into a file called test.log)

cd C:\Program Files\Java\jdk1.5.0_09\bin (this ensures you are in correct directory to launch java JVM)

java -cp ".;" com.util.cmmc.CopyAndDelete 1>>c:\test.log 2>>&1 (this is the key line that launches your app. In this case it's called CopyAndDelete. Classpath has no supporting classes besides default classes. Note that standard output statements are written to the same log file)
echo Batch file completed (just a comment)

6. point to batch file in Windows Scheduler. Open Windows Scheduler by Contol Panel | Scheduled Tasks (Windows 2003 Server Enterprise Ed.). Create a new task. Point to the .bat file you just created:

7. Configure Windows Scheduler to run on a schedule. Use Advanced settings if you need to further define scheduled run time.







Advanced settings allow repeating schedule and frequency not found in the basic settings:

Setup is now complete. You can do a sample run (independent of schedule) by right clicking on the Scheduled Task (at Explorer level - folder level) and clicking Run.

Note that there are many options for creating a .bat file, setting up scheduler, logging, etc. This blog entry shows how to do a basic, workable system from start-to-finish.  You can use this as a starting point and construct what you need from this example working system.

Objective: wish to have two object types and subtypes indexed only. Do not want dm_sysobject indexed, nor dm_document type. So this process includes full-text indexing for just certain types in the object model.

Since out of the box dm_sysobject is registered for full-text indexing, all objects under that type and subtypes will get full-text indexed, including dm_application, dm_job, job reports, and all their attributes.  However, usually you don’t wish to full-text index everything, just documents from selected types. This process accomplishes the goal of filtering what is included in the full-text index.

Procedure

Note: this is my rewritten procedure. Originally the basic process was an EMC/Documentum support note which was very poorly worded and took me hours to decipher.

Remember to try this in a sandbox or dev environment first. This process was implemented in two systems and then successfully put into a final production system.

In this example we want to register only two types & all their subtypes and attributes: rtd_document and dm_message_archive. One of these types is a custom type

In order to re-index only the objects from the two selected types follow these steps:

1. Unregister and register the super type that you want to register

- In iapi.exe unregister the current events for the dm_fulltext_index_user  for  dm_sysobject. You can use Documentum Administrator or iapi.exe.

Fetch r_object_id of dm_sysobject.. This is “0319978f80000105”  in this example. You could fetch using this query: select r_object_id from dm_type where name = 'dm_sysobject'

API> unregister,c,0319978f80000105,dm_save,dm_fulltext_index_user

API> unregister,c,0319978f80000105,dm_destroy,dm_fulltext_index_user

API> unregister,c,0319978f80000105,dm_readonlysave,dm_fulltext_index_user

API> unregister,c,0319978f80000105,dm_checkin,dm_fulltext_index_user

API> unregister,c,0319978f80000105,dm_move_content,dm_fulltext_index_user

1a. Logon to server via Remote Desktop and use iapi.exe. Or use local iapi.exe if you have installed locally.

Connect as dm_fulltext_index_user

API> getlogin,c,dm_fulltext_index_user,

Returns a dm_ticket with many characters as shown below. This is an example ticket only.

DM_TICKET=T0JKIE5VTEwgMAoxMwp2ZXJzaW9uIElOVCBTIDAKMwpmbGFncyBJTlQgUyAwCjAKc2VxdWVuY2VfbnVtIElOVCBTID

AKMzE3OQpjcmVhdGVfdGltZSBJTlQgUyAwCjEyMjU5OTIzNTgKZXhwaXJlX3RpbWUgSU5UIFMgMAoxMjI1OTkyNjU4CmRvbWFpbi

BTVFJJTkcgUyAwCkEgNyBydGQuZG9tCnVzZXJfbmFtZSBTVFJJTkcgUyAwCkEgMjIgZG1fZnVsbHRleHRfaW5kZXhfdXNlcgpwYXN

zd29yZCBTVFJJTkcgUyAwCkEgNzcgRE1fRU5DUl9URVhUPXdPU0RBNm1pNTgxajdiSmQxalltVWxmdUUvN0RpUEdVSFA3c0Yz

SWhGTXZlcjB3RFJyRVZtQmZiS0h5Um5rYXIKZG9jYmFzZV9uYW1lIFNUUklORyBTIDAKQSA0IGNFQ00KaG9zdF9uYW1lIFNUUklO

RyBTIDAKQSA4IGNDT05URU5UCnNlcnZlcl9uYW1lIFNUUklORyBTIDAKQSA0IGNFQ00Kc2lnbmF0dXJlX2xlbiBJTlQgUyAwCjU2C

nNpZ25hdHVyZSBTVFJJTkcgUyAwCkEgNTYgVnVSSHVqbWlVSytpaFpXa2UzTDVmTTB3WXp0bTBoZTZVV2w1eHJoRmhXUzZYY

UYxeWxScGpRPT0K

Using the login ticket connect as dm_fulltext_index_user. Paste the login ticket as shown below. Note that USER below in blue is a superuser, typically dm_admin or similar.

API> connect,USER,dm_fulltext_index_user,DM_TICKET=T0JKIE5VTEwgMAoxMwp2ZXJzaW9uIElOVCBTIDAKMwpmbGFncyBJTl

QgUyAwCjAKc2VxdWVuY2VfbnVtIElOVCBTIDAKMzE3OQpjcmVhdGVfdGltZSBJTlQgUyAwCjEyMjU5OTIzNTgKZXhwaXJlX3RpbWU

gSU5UIFMgMAoxMjI1OTkyNjU4CmRvbWFpbiBTVFJJTkcgUyAwCkEgNyBydGQuZG9tCnVzZXJfbmFtZSBTVFJJTkcgUyAwCkEgMjI

gZG1fZnVsbHRleHRfaW5kZXhfdXNlcgpwYXNzd29yZCBTVFJJTkcgUyAwCkEgNzcgRE1fRU5DUl9URVhUPXdPU0RBNm1pNTgx

ajdiSmQxalltVWxmdUUvN0RpUEdVSFA3c0YzSWhGTXZlcjB3RFJyRVZtQmZiS0h5Um5rYXIKZG9jYmFzZV9uYW1lIFNUUklORyBT

IDAKQSA0IGNFQ00KaG9zdF9uYW1lIFNUUklORyBTIDAKQSA4IGNDT05URU5UCnNlcnZlcl9uYW1lIFNUUklORyBTIDAKQSA0IG

NFQ00Kc2lnbmF0dXJlX2xlbiBJTlQgUyAwCjU2CnNpZ25hdHVyZSBTVFJJTkcgUyAwCkEgNTYgVnVSSHVqbWlVSytpaFpXa2UzTD

VmTTB3WXp0bTBoZTZVV2w1eHJoRmhXUzZYYUYxeWxScGpRPT0K

... Result:

s1

- Register events for desired types. Remember- we are registering just two supertypes and all their subtypes and attributes.  Run these steps first for rtd_document and repeat for dm_message_archive. Only rtd_document is shown in example below.

Fetch r_object_id for rtd_document…it is 030000fb800001d7.

Fetch r_object_id for dm_message_archive…it is 030000fb80000169.

API> register,c, 030000fb800001d7,dm_save,,F

SET> F

...

Returns: OK

API> register,c, 030000fb800001d7,dm_destroy,,F

SET> F

...

Returns: OK

API> register,c, 030000fb800001d7,dm_readonlysave,,F

SET> F

...

Returns: OK

API>register,c, 030000fb800001d7,dm_checkin,,F

SET> F

...

Returns: OK

API>register,c, 030000fb800001d7,dm_move_content,,F

SET> F

...

Returns: OK

2. Stop Index Agent. Do one time after registering dm_message_archive & rtd_document above.

3. From Fast InStream Admin page http://<index server>:16000. Go to collections and under "Collection Overview" find the name of the collection associated to the repository in question.

4. Click on the trash icon "Delete collection "<name of the collection>".

5. Wait until the process of deleting the collection completes:

- First it will delete the index information

- Second it will delete the fixml information

6. Using the Index Agent configuration program delete the Index Agent associated to the repository.

7. If the dm_ftengine_config and dm_ftindex_agent_config objects are still present, get the r_object_id and using iapi destory both objects:

API> retrieve,c,dm_ftengine_config

API> destroy,c,l

API> retrieve,c,dm_ftindex_agent_config

API> destroy,c,l

8. Run the Index Agent configuration program and reconfigure the Index Agent in normal mode. In other words make a new index agent.

9. Create an output file with the r_object_id's of the custom type. If two types (rtd_document & dm_message_archive) run twice: 1^st time run query for 1^st type, add these to ids.txt file.  Then run again for 2^nd type and append those to ids.txt file. Do this on Content Server.

idql -Uusername -Ppassword < dql_ statement_script > ids.txt

idql32 dev -Uuser -Ppassword -RsteveDQL.txt > c:\dqltest.txt

where dql_statement_script contains the select statement as follows:

dqltest.txt contains:

select r_object_id from rtd_document

go

- Open the output file and remove the headers and leave only the r_object_id values. Paste these id's into a file named ids.txt in directory listed below on cFast.

C:\Documentum\bea9.2\domains\DctmDomain\upload\IndexAgent1\IndexAgent1.war\WEB-INF\classes. 

File formerly called ids.txt is then change to ids.txt.done.  This is a good verification by FAST – it indicates object id’s were processed. The image below shows location of ids.txt after ids.txt is processed.

When you want to set up and use local logging for Java DFC classes here is an easy procedure.  Not only will you have local logging but after your code is deployed to application server (Java Method Server, Tomcat, WebLogic, etc.) the same logging setup can be configured there, too. An advantage is that you needn't use System.out statements locally and then change to log4j logging for server deployment but just configure once for log4j type logging.

This applies to Composer 6.5 but is probably back-compatible to other versions of Composer plus Eclipse 3x.

On your local developer's PC:

1. edit log4j.properties in c:\documentum\config or similar path. This assumes you have c:\documentum\config as a Linked Folder in your Composer project. See my other entries on setting up Composer for DFC projects.

2. add these lines or similar to log4j.properties file. See documentation on log4j logging formats if you wish to customize beyond what is shown below.

#----------------Custom Logging for Application called application_delete_notify -------------------------------------------

log4j.logger.application_delete_notify=DEBUG,application_delete_notify

log4j.appender.application_delete_notify=org.apache.log4j.RollingFileAppender

log4j.appender.application_delete_notify.File=c\:/Documentum/logs/application_delete_notify.log

log4j.appender.application_delete_notify.MaxFileSize=100MB

log4j.appender.application_delete_notify.layout=org.apache.log4j.PatternLayout

log4j.appender.application_delete_notify.layout.ConversionPattern=%d{HH\:mm\:ss} %p %c %m  %n

3. in your java classes be sure to import Logger:

import com.documentum.fc.common.DfLogger;

4. In your dfc code make sure you reference the logger with a statement something like this:

DfLogger.info("application_delete_notify", "DCTMLogin.doSession...session object is..." + sess,null,null);

Where:

 DfLogger.info = the logging level (could have used other logging levels like "debug")

 application_delete_notify = name of custom logger created in log4j.properties file

 DCTMLogin.doSession...session object is... = this is an arbitrary debugging string. You may put anything here. I wrote name of class, name of method, and outputting the hex value for a session object.

5.  After running code log file will appear in location you designated above on the third line of the log4j.properties file, in other words in this path:

c:/Documentum/logs/application_delete_notify.log

Documentum bloggers

• Laurence Hart - Word of Pie
• Lee Dallas and Marko Sillanpää - Big Men On Content
• http://dctmcontent.com - Documentum Articles & DQLs
• Robin East - Inside Documentum
• Johnney Gee - ASK JOHNNY! - DOCUMENTUM GURU
• Ajith Prabhakar -Ajith Prabhakar’s Weblog

Hard to believe it has come to this but Windows GUI searcher interfaces miss files frequently. I noticed this so many times that now I do command line searching...kind of sounds like a giant step backward? Now I'm confident of never missing any files. Windows command line searching is very good in these cases:

a. you really need to find a keyword inside a file because you're troubleshooting, debugging

b. you're coding and must locate a term inside a source file (.java, .vb, etc.)

To prove this point.... here is a case (Fig. 1) in which the search criteria included a search location of "hard drive and all directories and subdirectories". Files were not found even though there were two copies of the file present: one deep within an Eclipse directory and a copy on the desktop. Fig. 2 shows the results... the target files clearly contained the search term and is of type .java but files weren't found. Computer OS: XP Professional, Version 2002, SP 2. 

One Solution: 

How does command line search work?  Let's take a simple case - you just want to locate a file.  Use the "dir" command with the "/s" switch.

For example...command below searches in all subdirectories, beginning from current directory. So, if you're at the "root" of the "c" drive it will search for all documents containing SAX in the name for file type of ".java".  Pretty straightforward and covered years ago.

dir /s *SAX*.java 

What if you want to search for a string inside a file?  Also very easy, using the venerable "findstr" command. The variation below uses switch "/s" to search inside subdirectories, plus uses the "/m" switch to only show file names. If you don't use the "/m" switch it will show text of result file and location of the string in that file (which could be useful but I don't want in this particular search). See here for lots of details.

findstr /s /m "SAXBuilder" *.java

Results from this search found about 40 .java source files including the 2 target files I was looking for.  

This technique also works well for locating a string inside a log file (text file, MS Word, etc.) or similar task.  There may be a logical reason why Windows GUI search for OS cited above doesn't work properly. Personally, I don't really care because command line searching is not only totally reliable but is as fast, if not faster and one doesn't have all the annoying dogs to "help" you search!  In addition, in Vista OS you have a different search GUI.  If you use command line search you will always find targets regardless of Windows OS.  

Two items worth knowing: 1) usually you will not want to search all file types at the "c" root level because findstr command will attempt to open files that cannot be opened like Windows systems files; an error will be thrown plus the process may be really slowed down by attempting to open certain systems files.  Secondly, if you decide to abort a search (dir or findstr) at the command line, press Ctrl C twice.

Figure 1. Search criteria through GUI: 

 Figure 2. Results of search

What they don't tell you...  here's a way to find out about fast InStream administration, progress and processing using a web-based interface.  I sort of found this by accident. It wasn't mentioned in any Documentum documentation.  Note: be sure the correct service is started. By default this Administration panel is turned off as a Windows service. Service to start is FAST InStream Web Server.

Use this URL below. I ran from Remote Desktop login but it may not matter if you connect locally or not:

http://localhost:16000/admin/

This assumes a default setup with 16000 as port.  What you will see here is a great deal of information about the server to be used for monitoring and even configuration.  Areas include a Collection overview, System overview, quick log access and loads of other good stuff. Typically a Documentum administrator would not need to access this fast InStream Admin panel but if you have indexing problems (ha, ha) or have done custom configurations to fulltext indexes you may need to monitor/configure through this Admin acces point. 

The image in Fig. 1 gives some idea of what you see