What is the jewish court called?

It was called as Beth Din. In general Jewish laws and teachings discourage one Jew from suing another in a civil court. That's why rabbinical courts were made for that very purpose. The Jewish court has an orthodox method used by Jews whenever they have civil disputes.